You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Andre Reiter <a....@web.de> on 2011/07/19 09:59:33 UTC
HBase & MapReduce & Zookeeper
Hi folks,
i'm running in an interesting issue:
we have a zookeeper cluster running on 3 servers
we run mapreduce jobs using org.apache.hadoop.conf.Configuration to pass parameters to our mappers
the string based (key/value) approach is imho not the most elegant way, i would prefer to however pass ie a Writable object to my mappers, but however it works with the conf object, so what...
our jobs are using an hbase table as input (TableInputFormat)
the Configuration is created for every job, because we have every time new parameter values
new the problem occurs: the zookeeper cluster does not accept new connections, there is a limit per default to 10 connections for every client, and indeed all the old connections from the client to the zookeeper cluster are still established
new i found some description saying, that we have to reuse the Configuration object
ok, sure we can reuse the Configuration object, but if we want to set different parameters in a multi threaded environment, what than?
do i have to create a pool of Configuration objects, to share them synchronized?
andre
Re: HBase & MapReduce & Zookeeper
Posted by "Randy D. Wallace Jr." <ra...@randywallace.com>.
I had the same issue.
The problem for me turned out to be that the hbase.zookeeper.quorum was
not set in hbase-site.xml in the server that submitted the mapreduce
job. Ironically, this is also the same server that was running hbase
master. This defaulted to 127.0.0.1 which was where the tasktrackers
where trying to initiate the connection from. The fact that the
zookeeper quorum was not set in the tasktrackers was useless.
Re: HBase & MapReduce & Zookeeper
Posted by Andre Reiter <a....@web.de>.
hi Ruben, St.Ack
thanks a lot for your help!
finally, the problem seems to be solved by an pretty sick workaround
i did it like Bryan Keller described in this issue: https://issues.apache.org/jira/browse/HBASE-3792
@Ruben: thanks for the urls to that issues
cheers
andre
Re: HBase & MapReduce & Zookeeper
Posted by Stack <st...@duboce.net>.
Or override that method and in your version do not clone the Configuration.
St.Ack
On Thu, Jul 28, 2011 at 2:28 PM, Ruben Quintero <rf...@yahoo.com> wrote:
> Yes, that's the connection leak.
>
> Use deleteAllConnections(true), and it will close all open connections.
>
> - Ruben
>
>
>
>
> ________________________________
> From: Andre Reiter <a....@web.de>
> To: user@hbase.apache.org
> Sent: Thu, July 28, 2011 4:55:52 PM
> Subject: Re: HBase & MapReduce & Zookeeper
>
> i guess, i know the reason, why HConnectionManager.deleteConnection(conf,
> true); does not work for me
>
> in the MR job im using TableInputFormat, if you have a look at the source code
> in the method
> public void setConf(Configuration configuration)
> there is a line creating the HTable like this :
>
> ...
> setHTable(new HTable(new Configuration(conf), tableName));
> ...
>
> now the HTable is using a new configuration, why???
>
> HConnectionManager holds all Connections in a map with the config as a key:
> private static final Map<Configuration, HConnectionImplementation>
> HBASE_INSTANCE = ...
>
> Configuration does not override the equals and hash methods, so the HConnection
> created by the table using the new Configuration can not be referenced...
>
>
> just looking at executed code using debugger, i found out, that there is even
> one place more, where another new configuration is created and used:
> org.apache.hadoop.mapred.JobClient.submitJobInternal
>
> ...
> JobContext context = new JobContext(jobCopy, jobId);
> ...
>
> the JobContext constructor creates a new Configuration object: new
> org.apache.hadoop.mapred.JobConf(conf)
>
> so there is no real chance for me to get the configuration, which was used
> during the call to HConnectionManager.getConnection(conf)
>
>
>
> that is why HConnectionManager.deleteConnection(conf, true); does not work for
> me, or am i completely wrong???
>
>
> just going crazy with that! i'm not doing anything very special, all what i
> want, is to run a MR job out of my application, and NOT by calling it from the
> shell like this:
> ./bin/hadoop jar /tmp/my.jar package.OurClass
Re: HBase & MapReduce & Zookeeper
Posted by Ruben Quintero <rf...@yahoo.com>.
Yes, that's the connection leak.
Use deleteAllConnections(true), and it will close all open connections.
- Ruben
________________________________
From: Andre Reiter <a....@web.de>
To: user@hbase.apache.org
Sent: Thu, July 28, 2011 4:55:52 PM
Subject: Re: HBase & MapReduce & Zookeeper
i guess, i know the reason, why HConnectionManager.deleteConnection(conf,
true); does not work for me
in the MR job im using TableInputFormat, if you have a look at the source code
in the method
public void setConf(Configuration configuration)
there is a line creating the HTable like this :
...
setHTable(new HTable(new Configuration(conf), tableName));
...
now the HTable is using a new configuration, why???
HConnectionManager holds all Connections in a map with the config as a key:
private static final Map<Configuration, HConnectionImplementation>
HBASE_INSTANCE = ...
Configuration does not override the equals and hash methods, so the HConnection
created by the table using the new Configuration can not be referenced...
just looking at executed code using debugger, i found out, that there is even
one place more, where another new configuration is created and used:
org.apache.hadoop.mapred.JobClient.submitJobInternal
...
JobContext context = new JobContext(jobCopy, jobId);
...
the JobContext constructor creates a new Configuration object: new
org.apache.hadoop.mapred.JobConf(conf)
so there is no real chance for me to get the configuration, which was used
during the call to HConnectionManager.getConnection(conf)
that is why HConnectionManager.deleteConnection(conf, true); does not work for
me, or am i completely wrong???
just going crazy with that! i'm not doing anything very special, all what i
want, is to run a MR job out of my application, and NOT by calling it from the
shell like this:
./bin/hadoop jar /tmp/my.jar package.OurClass
Re: HBase & MapReduce & Zookeeper
Posted by Andre Reiter <a....@web.de>.
i guess, i know the reason, why HConnectionManager.deleteConnection(conf, true); does not work for me
in the MR job im using TableInputFormat, if you have a look at the source code in the method
public void setConf(Configuration configuration)
there is a line creating the HTable like this :
...
setHTable(new HTable(new Configuration(conf), tableName));
...
now the HTable is using a new configuration, why???
HConnectionManager holds all Connections in a map with the config as a key:
private static final Map<Configuration, HConnectionImplementation> HBASE_INSTANCE = ...
Configuration does not override the equals and hash methods, so the HConnection created by the table using the new Configuration can not be referenced...
just looking at executed code using debugger, i found out, that there is even one place more, where another new configuration is created and used:
org.apache.hadoop.mapred.JobClient.submitJobInternal
...
JobContext context = new JobContext(jobCopy, jobId);
...
the JobContext constructor creates a new Configuration object: new org.apache.hadoop.mapred.JobConf(conf)
so there is no real chance for me to get the configuration, which was used during the call to HConnectionManager.getConnection(conf)
that is why HConnectionManager.deleteConnection(conf, true); does not work for me, or am i completely wrong???
just going crazy with that! i'm not doing anything very special, all what i want, is to run a MR job out of my application, and NOT by calling it from the shell like this:
./bin/hadoop jar /tmp/my.jar package.OurClass
Re: HBase & MapReduce & Zookeeper
Posted by Ruben Quintero <rf...@yahoo.com>.
This problem has come up a few times. There are leaked connections in the TIF
See:
https://issues.apache.org/jira/browse/HBASE-3792
https://issues.apache.org/jira/browse/HBASE-3777
A quick and (very) dirty solution is to call deleteAllConnections(bool) at the
end of your MapReduce jobs, or periodically. If you have no other tables or
pools, etc. open, then no problem. If you do, they'll start throwing
IOExceptions, but you can re-instantiate them with a new config and then
continue as usual. (You do have to change the config or it'll simply grab the
closed, cached one from the HCM).
Another way: The leak comes from inside of TableInputFormat.setConf, where the
Configuration gets cloned (so then it's hash in the HCM is lost):
setHTable(new HTable(new Configuration(conf), tableName));
This is done to prevent changes to a config from affecting the job and
vice-versa. If you're 100% sure the config won't be modified, you could subclass
TIF to not make this copy.
For me, I didn't have extra tables hanging around, so I just blast them with
deleteAllConnections. :)
- Ruben
________________________________
From: Jeff Whiting <je...@qualtrics.com>
To: user@hbase.apache.org
Sent: Thu, July 28, 2011 12:10:16 PM
Subject: Re: HBase & MapReduce & Zookeeper
10 connection maximum is too low. It has been recommended to go up to as many
as 2000 connections
in the list. This doesn't fix your problem but is something you should probably
have in your
configuration.
~Jeff
On 7/28/2011 10:00 AM, Stack wrote:
> Try getting the ZooKeeperWatcher from the connection on your way out
> and explicitly shutdown the zk connection (see TestZooKeeper unit test
> for example).
> St.Ack
>
> On Thu, Jul 28, 2011 at 6:01 AM, Andre Reiter<a....@web.de> wrote:
>> this issue is still not resolved...
>>
>> unfortunatelly calling HConnectionManager.deleteConnection(conf, true);
>> after the MR job is finished, does not close the connection to the zookeeper
>> we have 3 zookeeper nodes
>> by default there is a limit of 10 connections allowed from a single client
>>
>> so after running 30 MR jobs scheduled by our application, we have 30
>> unclosed connections, trying to start a new MR job results in a failure, the
>> connection to the zookeeper ensamble is droped...
>>
>> the work around to restart the whole application after 30 MR jobss is not
>> very elegant... :-(
>>
>>
>>
--
Jeff Whiting
Qualtrics Senior Software Engineer
jeffw@qualtrics.com
Re: HBase & MapReduce & Zookeeper
Posted by Jeff Whiting <je...@qualtrics.com>.
10 connection maximum is too low. It has been recommended to go up to as many as 2000 connections
in the list. This doesn't fix your problem but is something you should probably have in your
configuration.
~Jeff
On 7/28/2011 10:00 AM, Stack wrote:
> Try getting the ZooKeeperWatcher from the connection on your way out
> and explicitly shutdown the zk connection (see TestZooKeeper unit test
> for example).
> St.Ack
>
> On Thu, Jul 28, 2011 at 6:01 AM, Andre Reiter<a....@web.de> wrote:
>> this issue is still not resolved...
>>
>> unfortunatelly calling HConnectionManager.deleteConnection(conf, true);
>> after the MR job is finished, does not close the connection to the zookeeper
>> we have 3 zookeeper nodes
>> by default there is a limit of 10 connections allowed from a single client
>>
>> so after running 30 MR jobs scheduled by our application, we have 30
>> unclosed connections, trying to start a new MR job results in a failure, the
>> connection to the zookeeper ensamble is droped...
>>
>> the work around to restart the whole application after 30 MR jobss is not
>> very elegant... :-(
>>
>>
>>
--
Jeff Whiting
Qualtrics Senior Software Engineer
jeffw@qualtrics.com
Re: HBase & MapReduce & Zookeeper
Posted by Stack <st...@duboce.net>.
Try getting the ZooKeeperWatcher from the connection on your way out
and explicitly shutdown the zk connection (see TestZooKeeper unit test
for example).
St.Ack
On Thu, Jul 28, 2011 at 6:01 AM, Andre Reiter <a....@web.de> wrote:
> this issue is still not resolved...
>
> unfortunatelly calling HConnectionManager.deleteConnection(conf, true);
> after the MR job is finished, does not close the connection to the zookeeper
> we have 3 zookeeper nodes
> by default there is a limit of 10 connections allowed from a single client
>
> so after running 30 MR jobs scheduled by our application, we have 30
> unclosed connections, trying to start a new MR job results in a failure, the
> connection to the zookeeper ensamble is droped...
>
> the work around to restart the whole application after 30 MR jobss is not
> very elegant... :-(
>
>
>
Re: HBase & MapReduce & Zookeeper
Posted by Andre Reiter <a....@web.de>.
this issue is still not resolved...
unfortunatelly calling HConnectionManager.deleteConnection(conf, true); after the MR job is finished, does not close the connection to the zookeeper
we have 3 zookeeper nodes
by default there is a limit of 10 connections allowed from a single client
so after running 30 MR jobs scheduled by our application, we have 30 unclosed connections, trying to start a new MR job results in a failure, the connection to the zookeeper ensamble is droped...
the work around to restart the whole application after 30 MR jobss is not very elegant... :-(
Re: HBase & MapReduce & Zookeeper
Posted by Stack <st...@duboce.net>.
My guess is that it needs to ask the master for the regions so it can
make the splits used by mapper tasks (to find master, needs to ask zk,
etc.). Check it out yourself under the mapreduce package?
St.Ack
On Wed, Jul 20, 2011 at 3:06 PM, Andre Reiter <a....@web.de> wrote:
> Hi Ted,
> thanks for the reply,
>
> at the moment i'm hust wondering, why the client creates a zookeeper
> connection at all
> all the client has to do, is to schedule a MR job, which is done by
> connecting to the jobtracker and to provide all the needed stuff, config,
> some extra resources in the distributed cache etc...
> but what is the reasen for creating a zookeeper connection by the client?
>
> andre
>
>
> Ted Yu wrote:
>>
>> This seems to be cdh related.
>>
>>>> either the map HConnectionManager.HBASE_INSTANCES does not contain the
>>
>> connection for the current config
>> You need to pass the same conf object.
>>
>> In trunk, I added the following:
>> public static void deleteStaleConnection(HConnection connection) {
>>
>> See
>>
>> http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
>>
>> Last resort is:
>> public static void deleteAllConnections(boolean stopProxy) {
>>
>> Cheers
>>
>
>
Re: HBase & MapReduce & Zookeeper
Posted by Andre Reiter <a....@web.de>.
Hi Ted,
thanks for the reply,
at the moment i'm hust wondering, why the client creates a zookeeper connection at all
all the client has to do, is to schedule a MR job, which is done by connecting to the jobtracker and to provide all the needed stuff, config, some extra resources in the distributed cache etc...
but what is the reasen for creating a zookeeper connection by the client?
andre
Ted Yu wrote:
> This seems to be cdh related.
>
>>> either the map HConnectionManager.HBASE_INSTANCES does not contain the
> connection for the current config
> You need to pass the same conf object.
>
> In trunk, I added the following:
> public static void deleteStaleConnection(HConnection connection) {
>
> See
> http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
>
> Last resort is:
> public static void deleteAllConnections(boolean stopProxy) {
>
> Cheers
>
Re: HBase & MapReduce & Zookeeper
Posted by Ted Yu <yu...@gmail.com>.
This seems to be cdh related.
>> either the map HConnectionManager.HBASE_INSTANCES does not contain the
connection for the current config
You need to pass the same conf object.
In trunk, I added the following:
public static void deleteStaleConnection(HConnection connection) {
See
http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
Last resort is:
public static void deleteAllConnections(boolean stopProxy) {
Cheers
On Wed, Jul 20, 2011 at 2:03 PM, Andre Reiter <a....@web.de> wrote:
> unfortunatelly there was no such LOG entry... :-(
>
> our versions:
> hadoop-0.20.2-CDH3B4
> hbase-0.90.1-CDH3B4
> zookeeper-3.3.2-CDH3B4
>
> either the map HConnectionManager.HBASE_**INSTANCES does not contain the
> connection for the current config, or HConnectionImplementation.**zooKeeper
> is null
> but the zookeeper connections are there, i can see them with "lsof | grep
> 2181"
>
> andre
>
>
>
> schrieb Ted Yu:
>
> Andre:
>> So you didn't see the following in client log (HConnectionManager line
>> 1067)
>> ?
>> LOG.info("Closed zookeeper sessionid=0x" +
>> Long.toHexString(this.**zooKeeper.getZooKeeper().**
>> getSessionId()));
>> HConnectionManager.**deleteConnection(conf, true) is supposed to close zk
>> connection in 0.90.x
>>
>>
>
Re: HBase & MapReduce & Zookeeper
Posted by Andre Reiter <a....@web.de>.
unfortunatelly there was no such LOG entry... :-(
our versions:
hadoop-0.20.2-CDH3B4
hbase-0.90.1-CDH3B4
zookeeper-3.3.2-CDH3B4
either the map HConnectionManager.HBASE_INSTANCES does not contain the connection for the current config, or HConnectionImplementation.zooKeeper is null
but the zookeeper connections are there, i can see them with "lsof | grep 2181"
andre
schrieb Ted Yu:
> Andre:
> So you didn't see the following in client log (HConnectionManager line 1067)
> ?
> LOG.info("Closed zookeeper sessionid=0x" +
> Long.toHexString(this.zooKeeper.getZooKeeper().getSessionId()));
> HConnectionManager.deleteConnection(conf, true) is supposed to close zk
> connection in 0.90.x
>
Re: HBase & MapReduce & Zookeeper
Posted by Ted Yu <yu...@gmail.com>.
Andre:
So you didn't see the following in client log (HConnectionManager line 1067)
?
LOG.info("Closed zookeeper sessionid=0x" +
Long.toHexString(this.zooKeeper.getZooKeeper().getSessionId()));
HConnectionManager.deleteConnection(conf, true) is supposed to close zk
connection in 0.90.x
On Wed, Jul 20, 2011 at 1:15 PM, Andre Reiter <a....@web.de> wrote:
> Hi St.Ack,
>
> actually calling HConnectionManager.**deleteConnection(conf, true); does
> not close the connection to the zookeeper
> i still can see the connection established...
>
> andre
>
>
>
>
> Stack wrote:
>
>> Then similarly, can you do the deleteConnection above in your client
>> or reuse the Configuration client-side that you use setting up the
>> job?
>> St.Ack
>>
>>
>
Re: HBase & MapReduce & Zookeeper
Posted by Andre Reiter <a....@web.de>.
Hi St.Ack,
actually calling HConnectionManager.deleteConnection(conf, true); does not close the connection to the zookeeper
i still can see the connection established...
andre
Stack wrote:
> Then similarly, can you do the deleteConnection above in your client
> or reuse the Configuration client-side that you use setting up the
> job?
> St.Ack
>
Re: HBase & MapReduce & Zookeeper
Posted by Stack <st...@duboce.net>.
Then similarly, can you do the deleteConnection above in your client
or reuse the Configuration client-side that you use setting up the
job?
St.Ack
On Wed, Jul 20, 2011 at 12:13 AM, Andre Reiter <a....@web.de> wrote:
> Hi Stack,
>
> just to make clear, actually the connections to the zookeeper being kept are
> not on our mappers (tasktrackers) but on the client, which schedules the MR
> job
> i think, the mappers are just fine, as they are
>
> andre
>
>
>
> Stack wrote:
>>
>> Can you reuse Configuration instances though the "configuration" changes?
>>
>> Else in your Mapper#cleanup, call HTable.close() then try
>> HConnectionManager.deleteConnection(table.getConfiguration()) after
>> close (could be issue with executors used by multi* operations not
>> completing before delete of connection finishes.
>>
>> St.Ack
>
>
Re: HBase & MapReduce & Zookeeper
Posted by Andre Reiter <a....@web.de>.
Hi Stack,
just to make clear, actually the connections to the zookeeper being kept are not on our mappers (tasktrackers) but on the client, which schedules the MR job
i think, the mappers are just fine, as they are
andre
Stack wrote:
> Can you reuse Configuration instances though the "configuration" changes?
>
> Else in your Mapper#cleanup, call HTable.close() then try
> HConnectionManager.deleteConnection(table.getConfiguration()) after
> close (could be issue with executors used by multi* operations not
> completing before delete of connection finishes.
>
> St.Ack
Re: HBase & MapReduce & Zookeeper
Posted by Stack <st...@duboce.net>.
Can you reuse Configuration instances though the "configuration" changes?
Else in your Mapper#cleanup, call HTable.close() then try
HConnectionManager.deleteConnection(table.getConfiguration()) after
close (could be issue with executors used by multi* operations not
completing before delete of connection finishes.
St.Ack
On Tue, Jul 19, 2011 at 11:36 PM, Andre Reiter <a....@web.de> wrote:
> Hi St.Ack,
>
> thanks for your reply
> but funally i miss the point, what would be the options to solve our issue?
>
> andre
>
>
Re: HBase & MapReduce & Zookeeper
Posted by Andre Reiter <a....@web.de>.
Hi St.Ack,
thanks for your reply
but funally i miss the point, what would be the options to solve our issue?
andre
Re: HBase & MapReduce & Zookeeper
Posted by Stack <st...@duboce.net>.
Configuration is not Comparable. Its instance identity that is used
comparing Configurations down in the guts of HConnectionManager in
0.90.x hbase so even if you reuse a Configuration and tweak it per
job, as far as HCM is concerned its the 'same'.
Are you seeing otherwise?
St.Ack
On Tue, Jul 19, 2011 at 5:40 AM, Andre Reiter <a....@web.de> wrote:
> Hi Doug,
>
> thanks a lot for reply,
> it's clear, that there is a parameter for maxClientCnxns, which is 10 by
> default
> of course i could increase it to s.th. big. but like i said, the old
> connections are still there, and i cannot imagine, that this is a correct
> behaviour, to let them open (established)
> so this is not a solution of the problem, but a delay
>
> either we must close the old zookeeper connections (i don't knnow how), or
> we must reuse them by reusing the Configuration object, which (like is said)
> is not that simple, due to different parameters for each MR job
>
>
Re: HBase & MapReduce & Zookeeper
Posted by Andre Reiter <a....@web.de>.
Hi Doug,
thanks a lot for reply,
it's clear, that there is a parameter for maxClientCnxns, which is 10 by default
of course i could increase it to s.th. big. but like i said, the old connections are still there, and i cannot imagine, that this is a correct behaviour, to let them open (established)
so this is not a solution of the problem, but a delay
either we must close the old zookeeper connections (i don't knnow how), or we must reuse them by reusing the Configuration object, which (like is said) is not that simple, due to different parameters for each MR job
Re: HBase & MapReduce & Zookeeper
Posted by Doug Meil <do...@explorysmedical.com>.
Hi there-
re: "that we have to reuse the Configuration object"
You are probably referring to this...
http://hbase.apache.org/book.html#client.connections
... yes, that is general guidance on client connection..
re: "do i have to create a pool of Configuration objects, to share them
synchronized?"
No.
re: "zookeeper cluster does not accept new connections"
http://zookeeper.apache.org/doc/r3.3.1/zookeeperAdmin.html
"maxClientCnxns The default is 10"
On 7/19/11 3:59 AM, "Andre Reiter" <a....@web.de> wrote:
>Hi folks,
>
>i'm running in an interesting issue:
>we have a zookeeper cluster running on 3 servers
>
>we run mapreduce jobs using org.apache.hadoop.conf.Configuration to pass
>parameters to our mappers
>the string based (key/value) approach is imho not the most elegant way, i
>would prefer to however pass ie a Writable object to my mappers, but
>however it works with the conf object, so what...
>
>our jobs are using an hbase table as input (TableInputFormat)
>the Configuration is created for every job, because we have every time
>new parameter values
>
>new the problem occurs: the zookeeper cluster does not accept new
>connections, there is a limit per default to 10 connections for every
>client, and indeed all the old connections from the client to the
>zookeeper cluster are still established
>
>new i found some description saying, that we have to reuse the
>Configuration object
>ok, sure we can reuse the Configuration object, but if we want to set
>different parameters in a multi threaded environment, what than?
>
>do i have to create a pool of Configuration objects, to share them
>synchronized?
>
>andre
>