You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by lohit <lo...@gmail.com> on 2011/04/29 04:26:30 UTC
HConnectionManager max cached instances
Hi,
By looking at HConnectionManager it looks like from a single node a max of
31 connections can be cached.
> static final int MAX_CACHED_HBASE_INSTANCES = 31;
I read the comment that this is based on the assumption that max client
connections to zookeeper is 30 from a single node.
ZooKeeper has an option to change that value, but we seem to be doing lot of
cache misses if we made more than 30 connections from a client.
This cache miss end up making new ZK connection and if number of HTable
operations are more, we see spike in ZooKeeper connections from single node.
This is seen in hbase.0.90.2. Is there any work around for this apart from
change the code?
--
Thanks
Lohit
Re: HConnectionManager max cached instances
Posted by Ted Yu <yu...@gmail.com>.
With the design from HBASE-3777, HTables would share connection.
On Thu, Apr 28, 2011 at 7:59 PM, M. C. Srivas <mc...@gmail.com> wrote:
> I am missing something here.
>
> For a given hbase client (ie, inside 1 JVM), shouldn't there be only 1
> connection to ZK, no matter how many htables are opened?
>
>
>
> On Thu, Apr 28, 2011 at 7:51 PM, lohit <lo...@gmail.com> wrote:
>
> > Thanks for the pointer. Even if we increase
> > hbase.zookeeper.property.maxClientCnxns
> > wouldn't the cache size still be 31?
> > I was wondering if there was way to avoid cache miss so that it would not
> > create new ZK connection.
> >
> > 2011/4/28 Ted Yu <yu...@gmail.com>
> >
> > > Increase value for the following parameter:
> > > <property>
> > > <name>hbase.zookeeper.property.maxClientCnxns</name>
> > > <value>30</value>
> > >
> > > Please read my blog for long-term fix which should be commited soon:
> > >
> > >
> >
> http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
> > >
> > > On Thu, Apr 28, 2011 at 7:26 PM, lohit <lo...@gmail.com>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > By looking at HConnectionManager it looks like from a single node a
> max
> > > of
> > > > 31 connections can be cached.
> > > >
> > > > > static final int MAX_CACHED_HBASE_INSTANCES = 31;
> > > >
> > > > I read the comment that this is based on the assumption that max
> client
> > > > connections to zookeeper is 30 from a single node.
> > > > ZooKeeper has an option to change that value, but we seem to be doing
> > lot
> > > > of
> > > > cache misses if we made more than 30 connections from a client.
> > > > This cache miss end up making new ZK connection and if number of
> HTable
> > > > operations are more, we see spike in ZooKeeper connections from
> single
> > > > node.
> > > >
> > > > This is seen in hbase.0.90.2. Is there any work around for this apart
> > > from
> > > > change the code?
> > > >
> > > > --
> > > > Thanks
> > > > Lohit
> > > >
> > >
> >
> >
> >
> > --
> > Have a Nice Day!
> > Lohit
> >
>
Re: HConnectionManager max cached instances
Posted by "M. C. Srivas" <mc...@gmail.com>.
I am missing something here.
For a given hbase client (ie, inside 1 JVM), shouldn't there be only 1
connection to ZK, no matter how many htables are opened?
On Thu, Apr 28, 2011 at 7:51 PM, lohit <lo...@gmail.com> wrote:
> Thanks for the pointer. Even if we increase
> hbase.zookeeper.property.maxClientCnxns
> wouldn't the cache size still be 31?
> I was wondering if there was way to avoid cache miss so that it would not
> create new ZK connection.
>
> 2011/4/28 Ted Yu <yu...@gmail.com>
>
> > Increase value for the following parameter:
> > <property>
> > <name>hbase.zookeeper.property.maxClientCnxns</name>
> > <value>30</value>
> >
> > Please read my blog for long-term fix which should be commited soon:
> >
> >
> http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
> >
> > On Thu, Apr 28, 2011 at 7:26 PM, lohit <lo...@gmail.com>
> wrote:
> >
> > > Hi,
> > >
> > > By looking at HConnectionManager it looks like from a single node a max
> > of
> > > 31 connections can be cached.
> > >
> > > > static final int MAX_CACHED_HBASE_INSTANCES = 31;
> > >
> > > I read the comment that this is based on the assumption that max client
> > > connections to zookeeper is 30 from a single node.
> > > ZooKeeper has an option to change that value, but we seem to be doing
> lot
> > > of
> > > cache misses if we made more than 30 connections from a client.
> > > This cache miss end up making new ZK connection and if number of HTable
> > > operations are more, we see spike in ZooKeeper connections from single
> > > node.
> > >
> > > This is seen in hbase.0.90.2. Is there any work around for this apart
> > from
> > > change the code?
> > >
> > > --
> > > Thanks
> > > Lohit
> > >
> >
>
>
>
> --
> Have a Nice Day!
> Lohit
>
Re: HConnectionManager max cached instances
Posted by Ted Yu <yu...@gmail.com>.
Pardon my limited knowledge of asynchbase internals, Benoit.
HBASE-3777 is big. It is not specific to RegionServer operations.
It is now easy to see a good design (M. C.
Srivas<https://issues.apache.org/jira/secure/ViewProfile.jspa?name=srivas>disagrees)
which wasn't in 0.90 or trunk.
w.r.t. M. C. Srivas<https://issues.apache.org/jira/secure/ViewProfile.jspa?name=srivas>'s
comments, the way connection-specific properties were chosen has historical
reason. It has to cover all use cases from both server, client and unit test
code where developer(s) expected new connection to be made.
One of my initial goals was to make TableOutputFormat more user friendly.
After HBASE-3777 gets checked in, we will continue listening to user
feedback and see if my goal has been achieved.
Thanks
On Fri, Apr 29, 2011 at 12:56 AM, tsuna <ts...@gmail.com> wrote:
> On Thu, Apr 28, 2011 at 7:59 PM, Ted Yu <yu...@gmail.com> wrote:
> > I would be happy to know that solution.
> > If that solution exists, Karthick and I wouldn't have spent 2 weeks on
> > HBASE-3777 :-)
>
> I haven't had time to read through this gigantic issue, so sorry if my
> question is stupid. Is this problem specific to the RegionServer, or
> does it also affect regular HBase clients? Your blog post refers to
> HBASE-3734, which seems specific to RegionServer operations, but I see
> that the v6 of the patch on HBASE-3777 also touches a lot of code that
> relates to regular HBase clients.
>
> I'm having trouble understanding why this is such a big issue.
> asynchbase maintains exactly one connection per RegionServer you need
> to interact with, and it does NOT maintain any ZooKeeper connection at
> all, except when it needs to find -ROOT- (which typically happens only
> when your application starts, and then very rarely as servers hosting
> .META. and -ROOT- fail).
>
> --
> Benoit "tsuna" Sigoure
> Software Engineer @ www.StumbleUpon.com
>
Re: HConnectionManager max cached instances
Posted by tsuna <ts...@gmail.com>.
On Thu, Apr 28, 2011 at 7:59 PM, Ted Yu <yu...@gmail.com> wrote:
> I would be happy to know that solution.
> If that solution exists, Karthick and I wouldn't have spent 2 weeks on
> HBASE-3777 :-)
I haven't had time to read through this gigantic issue, so sorry if my
question is stupid. Is this problem specific to the RegionServer, or
does it also affect regular HBase clients? Your blog post refers to
HBASE-3734, which seems specific to RegionServer operations, but I see
that the v6 of the patch on HBASE-3777 also touches a lot of code that
relates to regular HBase clients.
I'm having trouble understanding why this is such a big issue.
asynchbase maintains exactly one connection per RegionServer you need
to interact with, and it does NOT maintain any ZooKeeper connection at
all, except when it needs to find -ROOT- (which typically happens only
when your application starts, and then very rarely as servers hosting
.META. and -ROOT- fail).
--
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com
Re: HConnectionManager max cached instances
Posted by lohit <lo...@gmail.com>.
2011/4/28 Ted Yu <yu...@gmail.com>
> I would be happy to know that solution.
> If that solution exists, Karthick and I wouldn't have spent 2 weeks on
> HBASE-3777 :-)
>
I was about to suggest that the size be set to same as maxClientConnections,
but I see that it has already been done in HBASE-3777
>
> On Thu, Apr 28, 2011 at 7:51 PM, lohit <lo...@gmail.com> wrote:
>
> > Thanks for the pointer. Even if we increase
> > hbase.zookeeper.property.maxClientCnxns
> > wouldn't the cache size still be 31?
> > I was wondering if there was way to avoid cache miss so that it would not
> > create new ZK connection.
> >
> > 2011/4/28 Ted Yu <yu...@gmail.com>
> >
> > > Increase value for the following parameter:
> > > <property>
> > > <name>hbase.zookeeper.property.maxClientCnxns</name>
> > > <value>30</value>
> > >
> > > Please read my blog for long-term fix which should be commited soon:
> > >
> > >
> >
> http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
> > >
> > > On Thu, Apr 28, 2011 at 7:26 PM, lohit <lo...@gmail.com>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > By looking at HConnectionManager it looks like from a single node a
> max
> > > of
> > > > 31 connections can be cached.
> > > >
> > > > > static final int MAX_CACHED_HBASE_INSTANCES = 31;
> > > >
> > > > I read the comment that this is based on the assumption that max
> client
> > > > connections to zookeeper is 30 from a single node.
> > > > ZooKeeper has an option to change that value, but we seem to be doing
> > lot
> > > > of
> > > > cache misses if we made more than 30 connections from a client.
> > > > This cache miss end up making new ZK connection and if number of
> HTable
> > > > operations are more, we see spike in ZooKeeper connections from
> single
> > > > node.
> > > >
> > > > This is seen in hbase.0.90.2. Is there any work around for this apart
> > > from
> > > > change the code?
> > > >
> > > > --
> > > > Thanks
> > > > Lohit
> > > >
> > >
> >
> >
> >
> > --
> > Have a Nice Day!
> > Lohit
> >
>
--
Have a Nice Day!
Lohit
Re: HConnectionManager max cached instances
Posted by Ted Yu <yu...@gmail.com>.
I would be happy to know that solution.
If that solution exists, Karthick and I wouldn't have spent 2 weeks on
HBASE-3777 :-)
On Thu, Apr 28, 2011 at 7:51 PM, lohit <lo...@gmail.com> wrote:
> Thanks for the pointer. Even if we increase
> hbase.zookeeper.property.maxClientCnxns
> wouldn't the cache size still be 31?
> I was wondering if there was way to avoid cache miss so that it would not
> create new ZK connection.
>
> 2011/4/28 Ted Yu <yu...@gmail.com>
>
> > Increase value for the following parameter:
> > <property>
> > <name>hbase.zookeeper.property.maxClientCnxns</name>
> > <value>30</value>
> >
> > Please read my blog for long-term fix which should be commited soon:
> >
> >
> http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
> >
> > On Thu, Apr 28, 2011 at 7:26 PM, lohit <lo...@gmail.com>
> wrote:
> >
> > > Hi,
> > >
> > > By looking at HConnectionManager it looks like from a single node a max
> > of
> > > 31 connections can be cached.
> > >
> > > > static final int MAX_CACHED_HBASE_INSTANCES = 31;
> > >
> > > I read the comment that this is based on the assumption that max client
> > > connections to zookeeper is 30 from a single node.
> > > ZooKeeper has an option to change that value, but we seem to be doing
> lot
> > > of
> > > cache misses if we made more than 30 connections from a client.
> > > This cache miss end up making new ZK connection and if number of HTable
> > > operations are more, we see spike in ZooKeeper connections from single
> > > node.
> > >
> > > This is seen in hbase.0.90.2. Is there any work around for this apart
> > from
> > > change the code?
> > >
> > > --
> > > Thanks
> > > Lohit
> > >
> >
>
>
>
> --
> Have a Nice Day!
> Lohit
>
re: HConnectionManager max cached instances
Posted by Gaojinchao <ga...@huawei.com>.
In version 0.90.3, It has set for 2001
-----邮件原件-----
发件人: lohit [mailto:lohit.vijayarenu@gmail.com]
发送时间: 2011年4月29日 10:52
收件人: dev@hbase.apache.org
主题: Re: HConnectionManager max cached instances
Thanks for the pointer. Even if we increase
hbase.zookeeper.property.maxClientCnxns
wouldn't the cache size still be 31?
I was wondering if there was way to avoid cache miss so that it would not
create new ZK connection.
2011/4/28 Ted Yu <yu...@gmail.com>
> Increase value for the following parameter:
> <property>
> <name>hbase.zookeeper.property.maxClientCnxns</name>
> <value>30</value>
>
> Please read my blog for long-term fix which should be commited soon:
>
> http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
>
> On Thu, Apr 28, 2011 at 7:26 PM, lohit <lo...@gmail.com> wrote:
>
> > Hi,
> >
> > By looking at HConnectionManager it looks like from a single node a max
> of
> > 31 connections can be cached.
> >
> > > static final int MAX_CACHED_HBASE_INSTANCES = 31;
> >
> > I read the comment that this is based on the assumption that max client
> > connections to zookeeper is 30 from a single node.
> > ZooKeeper has an option to change that value, but we seem to be doing lot
> > of
> > cache misses if we made more than 30 connections from a client.
> > This cache miss end up making new ZK connection and if number of HTable
> > operations are more, we see spike in ZooKeeper connections from single
> > node.
> >
> > This is seen in hbase.0.90.2. Is there any work around for this apart
> from
> > change the code?
> >
> > --
> > Thanks
> > Lohit
> >
>
--
Have a Nice Day!
Lohit
Re: HConnectionManager max cached instances
Posted by lohit <lo...@gmail.com>.
Thanks for the pointer. Even if we increase
hbase.zookeeper.property.maxClientCnxns
wouldn't the cache size still be 31?
I was wondering if there was way to avoid cache miss so that it would not
create new ZK connection.
2011/4/28 Ted Yu <yu...@gmail.com>
> Increase value for the following parameter:
> <property>
> <name>hbase.zookeeper.property.maxClientCnxns</name>
> <value>30</value>
>
> Please read my blog for long-term fix which should be commited soon:
>
> http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
>
> On Thu, Apr 28, 2011 at 7:26 PM, lohit <lo...@gmail.com> wrote:
>
> > Hi,
> >
> > By looking at HConnectionManager it looks like from a single node a max
> of
> > 31 connections can be cached.
> >
> > > static final int MAX_CACHED_HBASE_INSTANCES = 31;
> >
> > I read the comment that this is based on the assumption that max client
> > connections to zookeeper is 30 from a single node.
> > ZooKeeper has an option to change that value, but we seem to be doing lot
> > of
> > cache misses if we made more than 30 connections from a client.
> > This cache miss end up making new ZK connection and if number of HTable
> > operations are more, we see spike in ZooKeeper connections from single
> > node.
> >
> > This is seen in hbase.0.90.2. Is there any work around for this apart
> from
> > change the code?
> >
> > --
> > Thanks
> > Lohit
> >
>
--
Have a Nice Day!
Lohit
Re: HConnectionManager max cached instances
Posted by Ted Yu <yu...@gmail.com>.
Increase value for the following parameter:
<property>
<name>hbase.zookeeper.property.maxClientCnxns</name>
<value>30</value>
Please read my blog for long-term fix which should be commited soon:
http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
On Thu, Apr 28, 2011 at 7:26 PM, lohit <lo...@gmail.com> wrote:
> Hi,
>
> By looking at HConnectionManager it looks like from a single node a max of
> 31 connections can be cached.
>
> > static final int MAX_CACHED_HBASE_INSTANCES = 31;
>
> I read the comment that this is based on the assumption that max client
> connections to zookeeper is 30 from a single node.
> ZooKeeper has an option to change that value, but we seem to be doing lot
> of
> cache misses if we made more than 30 connections from a client.
> This cache miss end up making new ZK connection and if number of HTable
> operations are more, we see spike in ZooKeeper connections from single
> node.
>
> This is seen in hbase.0.90.2. Is there any work around for this apart from
> change the code?
>
> --
> Thanks
> Lohit
>