You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Mark Vigeant <ma...@riskmetrics.com> on 2009/10/06 16:57:20 UTC

NoNode Exception?

Hey-

So I've been messing around with HBase 0.20.0 for the past couple of weeks on some virtual machines (with ubuntu), and for whatever reason every week or so I get thrown a ton of exceptions and have no idea what to do except format the nodes and reinstall everything. Basically I get this :

org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/master

but it's silly because I know the master is running (when typing jps it says HMaster and HQuorumPeer are running fine) so I was wondering if there was a patch or something that fixes this... I don't think this is because of something I did because it occurs randomly. I don't know. I throw myself at the mercy of the listserve.

Mark Vigeant
RiskMetrics Group, Inc.

RE: NoNode Exception?

Posted by Mark Vigeant <ma...@riskmetrics.com>.
That may very well be the problem. Thank you!

-----Original Message-----
From: Eric Bowman [mailto:ebowman@boboco.ie] 
Sent: Tuesday, October 06, 2009 11:10 AM
To: hbase-user@hadoop.apache.org
Subject: Re: NoNode Exception?

One thing that could cause this, is if you have a zookeeper ensemble in
which the individual zookeepers are running in stand-alone mode.

Mark Vigeant wrote:
> Hey-
>
> So I've been messing around with HBase 0.20.0 for the past couple of weeks on some virtual machines (with ubuntu), and for whatever reason every week or so I get thrown a ton of exceptions and have no idea what to do except format the nodes and reinstall everything. Basically I get this :
>
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/master
>
> but it's silly because I know the master is running (when typing jps it says HMaster and HQuorumPeer are running fine) so I was wondering if there was a patch or something that fixes this... I don't think this is because of something I did because it occurs randomly. I don't know. I throw myself at the mercy of the listserve.
>
> Mark Vigeant
> RiskMetrics Group, Inc.
>
>
>   


-- 
Eric Bowman
Boboco Ltd
ebowman@boboco.ie
http://www.boboco.ie/ebowman/pubkey.pgp
+35318394189/+353872801532

Re: NoNode Exception?

Posted by Eric Bowman <eb...@boboco.ie>.
One thing that could cause this, is if you have a zookeeper ensemble in
which the individual zookeepers are running in stand-alone mode.

Mark Vigeant wrote:
> Hey-
>
> So I've been messing around with HBase 0.20.0 for the past couple of weeks on some virtual machines (with ubuntu), and for whatever reason every week or so I get thrown a ton of exceptions and have no idea what to do except format the nodes and reinstall everything. Basically I get this :
>
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/master
>
> but it's silly because I know the master is running (when typing jps it says HMaster and HQuorumPeer are running fine) so I was wondering if there was a patch or something that fixes this... I don't think this is because of something I did because it occurs randomly. I don't know. I throw myself at the mercy of the listserve.
>
> Mark Vigeant
> RiskMetrics Group, Inc.
>
>
>   


-- 
Eric Bowman
Boboco Ltd
ebowman@boboco.ie
http://www.boboco.ie/ebowman/pubkey.pgp
+35318394189/+353872801532

RE: NoNode Exception?

Posted by Mark Vigeant <ma...@riskmetrics.com>.
Ok so I was just trying to start HBase up again so I could compile a list of error messages to send you when miraculously everything started working...

Maybe I fixed it? If I run into the problems again I'll send you the errors. Thank you so much for your immediate help though!

-----Original Message-----
From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Daniel Cryans
Sent: Tuesday, October 06, 2009 11:54 AM
To: hbase-user@hadoop.apache.org
Subject: Re: NoNode Exception?

Care to show us your configuration? Are you managing the zookeeper
ensemble yourself or do you let HBase do it?

Can I see a full stack trace along with some lines before and after?
Try to get the first one, when things go awry.

J-D

On Tue, Oct 6, 2009 at 11:49 AM, Mark Vigeant
<ma...@riskmetrics.com> wrote:
> Yeah I made a data directory for zookeeper, so that can't be it.
>
> And in the logs all I see are NoNodeExceptions. I reconfigured it so that zookeeper was no longer in standalone mode but the error is still occurring. Perhaps I have to reformat everything then configure it in distributed mode?
>
> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Daniel Cryans
> Sent: Tuesday, October 06, 2009 11:06 AM
> To: hbase-user@hadoop.apache.org
> Subject: Re: NoNode Exception?
>
> Mark,
>
> By default the data is stored in /tmp which is cleared regularly, did
> you change hbase.tmp.dir or hbase.zookeeper.property.dataDir?
>
> Also do you get other exceptions in the master and regionserver logs?
>
> J-D
>
> On Tue, Oct 6, 2009 at 10:57 AM, Mark Vigeant
> <ma...@riskmetrics.com> wrote:
>> Hey-
>>
>> So I've been messing around with HBase 0.20.0 for the past couple of weeks on some virtual machines (with ubuntu), and for whatever reason every week or so I get thrown a ton of exceptions and have no idea what to do except format the nodes and reinstall everything. Basically I get this :
>>
>> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/master
>>
>> but it's silly because I know the master is running (when typing jps it says HMaster and HQuorumPeer are running fine) so I was wondering if there was a patch or something that fixes this... I don't think this is because of something I did because it occurs randomly. I don't know. I throw myself at the mercy of the listserve.
>>
>> Mark Vigeant
>> RiskMetrics Group, Inc.
>>
>>
>

Re: NoNode Exception?

Posted by Eric Bowman <eb...@boboco.ie>.
I'd look in the zookeeper logs for more clues. We saw something similar
to this, and found OutOfMemoryErrors in the ZK logs.

Mark Vigeant wrote:
> So I said I'd tell you if the problem came up again, so here's the error that gets thrown at me over and over again:
>
> 09/10/02 23:59:23 WARN zookeeper.ClientCnxn: Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@92eb76
> java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> 	at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
> 09/10/02 23:59:23 WARN zookeeper.ClientCnxn: Ignoring exception during shutdown input
> java.net.SocketException: Transport endpoint is not connected
> 	at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> 	at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922
>
> This is actually different from what it was the other day. Any clue what's going on here?
>
> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Daniel Cryans
> Sent: Tuesday, October 06, 2009 11:54 AM
> To: hbase-user@hadoop.apache.org
> Subject: Re: NoNode Exception?
>
> Care to show us your configuration? Are you managing the zookeeper
> ensemble yourself or do you let HBase do it?
>
> Can I see a full stack trace along with some lines before and after?
> Try to get the first one, when things go awry.
>
> J-D
>
> On Tue, Oct 6, 2009 at 11:49 AM, Mark Vigeant
> <ma...@riskmetrics.com> wrote:
>   
>> Yeah I made a data directory for zookeeper, so that can't be it.
>>
>> And in the logs all I see are NoNodeExceptions. I reconfigured it so that zookeeper was no longer in standalone mode but the error is still occurring. Perhaps I have to reformat everything then configure it in distributed mode?
>>
>> -----Original Message-----
>> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Daniel Cryans
>> Sent: Tuesday, October 06, 2009 11:06 AM
>> To: hbase-user@hadoop.apache.org
>> Subject: Re: NoNode Exception?
>>
>> Mark,
>>
>> By default the data is stored in /tmp which is cleared regularly, did
>> you change hbase.tmp.dir or hbase.zookeeper.property.dataDir?
>>
>> Also do you get other exceptions in the master and regionserver logs?
>>
>> J-D
>>
>> On Tue, Oct 6, 2009 at 10:57 AM, Mark Vigeant
>> <ma...@riskmetrics.com> wrote:
>>     
>>> Hey-
>>>
>>> So I've been messing around with HBase 0.20.0 for the past couple of weeks on some virtual machines (with ubuntu), and for whatever reason every week or so I get thrown a ton of exceptions and have no idea what to do except format the nodes and reinstall everything. Basically I get this :
>>>
>>> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/master
>>>
>>> but it's silly because I know the master is running (when typing jps it says HMaster and HQuorumPeer are running fine) so I was wondering if there was a patch or something that fixes this... I don't think this is because of something I did because it occurs randomly. I don't know. I throw myself at the mercy of the listserve.
>>>
>>> Mark Vigeant
>>> RiskMetrics Group, Inc.
>>>
>>>
>>>       


-- 
Eric Bowman
Boboco Ltd
ebowman@boboco.ie
http://www.boboco.ie/ebowman/pubkey.pgp
+35318394189/+353872801532

RE: NoNode Exception?

Posted by Mark Vigeant <ma...@riskmetrics.com>.
So I said I'd tell you if the problem came up again, so here's the error that gets thrown at me over and over again:

09/10/02 23:59:23 WARN zookeeper.ClientCnxn: Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@92eb76
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
	at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
09/10/02 23:59:23 WARN zookeeper.ClientCnxn: Ignoring exception during shutdown input
java.net.SocketException: Transport endpoint is not connected
	at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
	at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
	at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
	at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922

This is actually different from what it was the other day. Any clue what's going on here?

-----Original Message-----
From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Daniel Cryans
Sent: Tuesday, October 06, 2009 11:54 AM
To: hbase-user@hadoop.apache.org
Subject: Re: NoNode Exception?

Care to show us your configuration? Are you managing the zookeeper
ensemble yourself or do you let HBase do it?

Can I see a full stack trace along with some lines before and after?
Try to get the first one, when things go awry.

J-D

On Tue, Oct 6, 2009 at 11:49 AM, Mark Vigeant
<ma...@riskmetrics.com> wrote:
> Yeah I made a data directory for zookeeper, so that can't be it.
>
> And in the logs all I see are NoNodeExceptions. I reconfigured it so that zookeeper was no longer in standalone mode but the error is still occurring. Perhaps I have to reformat everything then configure it in distributed mode?
>
> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Daniel Cryans
> Sent: Tuesday, October 06, 2009 11:06 AM
> To: hbase-user@hadoop.apache.org
> Subject: Re: NoNode Exception?
>
> Mark,
>
> By default the data is stored in /tmp which is cleared regularly, did
> you change hbase.tmp.dir or hbase.zookeeper.property.dataDir?
>
> Also do you get other exceptions in the master and regionserver logs?
>
> J-D
>
> On Tue, Oct 6, 2009 at 10:57 AM, Mark Vigeant
> <ma...@riskmetrics.com> wrote:
>> Hey-
>>
>> So I've been messing around with HBase 0.20.0 for the past couple of weeks on some virtual machines (with ubuntu), and for whatever reason every week or so I get thrown a ton of exceptions and have no idea what to do except format the nodes and reinstall everything. Basically I get this :
>>
>> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/master
>>
>> but it's silly because I know the master is running (when typing jps it says HMaster and HQuorumPeer are running fine) so I was wondering if there was a patch or something that fixes this... I don't think this is because of something I did because it occurs randomly. I don't know. I throw myself at the mercy of the listserve.
>>
>> Mark Vigeant
>> RiskMetrics Group, Inc.
>>
>>
>

Re: NoNode Exception?

Posted by Jean-Daniel Cryans <jd...@apache.org>.
Care to show us your configuration? Are you managing the zookeeper
ensemble yourself or do you let HBase do it?

Can I see a full stack trace along with some lines before and after?
Try to get the first one, when things go awry.

J-D

On Tue, Oct 6, 2009 at 11:49 AM, Mark Vigeant
<ma...@riskmetrics.com> wrote:
> Yeah I made a data directory for zookeeper, so that can't be it.
>
> And in the logs all I see are NoNodeExceptions. I reconfigured it so that zookeeper was no longer in standalone mode but the error is still occurring. Perhaps I have to reformat everything then configure it in distributed mode?
>
> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Daniel Cryans
> Sent: Tuesday, October 06, 2009 11:06 AM
> To: hbase-user@hadoop.apache.org
> Subject: Re: NoNode Exception?
>
> Mark,
>
> By default the data is stored in /tmp which is cleared regularly, did
> you change hbase.tmp.dir or hbase.zookeeper.property.dataDir?
>
> Also do you get other exceptions in the master and regionserver logs?
>
> J-D
>
> On Tue, Oct 6, 2009 at 10:57 AM, Mark Vigeant
> <ma...@riskmetrics.com> wrote:
>> Hey-
>>
>> So I've been messing around with HBase 0.20.0 for the past couple of weeks on some virtual machines (with ubuntu), and for whatever reason every week or so I get thrown a ton of exceptions and have no idea what to do except format the nodes and reinstall everything. Basically I get this :
>>
>> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/master
>>
>> but it's silly because I know the master is running (when typing jps it says HMaster and HQuorumPeer are running fine) so I was wondering if there was a patch or something that fixes this... I don't think this is because of something I did because it occurs randomly. I don't know. I throw myself at the mercy of the listserve.
>>
>> Mark Vigeant
>> RiskMetrics Group, Inc.
>>
>>
>

RE: NoNode Exception?

Posted by Mark Vigeant <ma...@riskmetrics.com>.
Yeah I made a data directory for zookeeper, so that can't be it.

And in the logs all I see are NoNodeExceptions. I reconfigured it so that zookeeper was no longer in standalone mode but the error is still occurring. Perhaps I have to reformat everything then configure it in distributed mode?

-----Original Message-----
From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Daniel Cryans
Sent: Tuesday, October 06, 2009 11:06 AM
To: hbase-user@hadoop.apache.org
Subject: Re: NoNode Exception?

Mark,

By default the data is stored in /tmp which is cleared regularly, did
you change hbase.tmp.dir or hbase.zookeeper.property.dataDir?

Also do you get other exceptions in the master and regionserver logs?

J-D

On Tue, Oct 6, 2009 at 10:57 AM, Mark Vigeant
<ma...@riskmetrics.com> wrote:
> Hey-
>
> So I've been messing around with HBase 0.20.0 for the past couple of weeks on some virtual machines (with ubuntu), and for whatever reason every week or so I get thrown a ton of exceptions and have no idea what to do except format the nodes and reinstall everything. Basically I get this :
>
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/master
>
> but it's silly because I know the master is running (when typing jps it says HMaster and HQuorumPeer are running fine) so I was wondering if there was a patch or something that fixes this... I don't think this is because of something I did because it occurs randomly. I don't know. I throw myself at the mercy of the listserve.
>
> Mark Vigeant
> RiskMetrics Group, Inc.
>
>

Re: NoNode Exception?

Posted by Jean-Daniel Cryans <jd...@apache.org>.
Mark,

By default the data is stored in /tmp which is cleared regularly, did
you change hbase.tmp.dir or hbase.zookeeper.property.dataDir?

Also do you get other exceptions in the master and regionserver logs?

J-D

On Tue, Oct 6, 2009 at 10:57 AM, Mark Vigeant
<ma...@riskmetrics.com> wrote:
> Hey-
>
> So I've been messing around with HBase 0.20.0 for the past couple of weeks on some virtual machines (with ubuntu), and for whatever reason every week or so I get thrown a ton of exceptions and have no idea what to do except format the nodes and reinstall everything. Basically I get this :
>
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/master
>
> but it's silly because I know the master is running (when typing jps it says HMaster and HQuorumPeer are running fine) so I was wondering if there was a patch or something that fixes this... I don't think this is because of something I did because it occurs randomly. I don't know. I throw myself at the mercy of the listserve.
>
> Mark Vigeant
> RiskMetrics Group, Inc.
>
>