You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Vaibhav Puranik <vp...@gmail.com> on 2009/07/24 19:48:00 UTC
Latest trunk - client throwing org.apache.hadoop.hbase.client.NoServerForRegionException:
Timed out trying to locate root region
Hi,
We have deployed the latest trunk (yesterday evening 5 pm) on EC2
1 master, 3 region servers.
This morning, when I am trying to run client script for querying the table,
I am getting following exception:
Caught: org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out
trying to locate root region
Any ideas, suggestions?
Regards,
Vaibhav Puranik
Gumgum
Re: Latest trunk - client throwing org.apache.hadoop.hbase.client.NoServerForRegionException:
Timed out trying to locate root region
Posted by stack <st...@duboce.net>.
Committed as https://issues.apache.org/jira/browse/HBASE-1704 (with spelling
correction -- thanks).
St.Ack
On Fri, Jul 24, 2009 at 5:45 PM, Ken Weiner <ke...@gumgum.com> wrote:
> St.Ack,
>
> That message looks great. I bet it will help a lot of people who
> misconfigure things.
> One minor spelling correction: "currenty=localhost:2181" should be
> "currently=localhost:2181"
>
> -Ken
>
> On Fri, Jul 24, 2009 at 2:49 PM, stack <st...@duboce.net> wrote:
>
> > Hows this:
> >
> > Index: src/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWrapper.java
> > ===================================================================
> > --- src/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWrapper.java
> > (revision 797665)
> > +++ src/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWrapper.java
> > (working copy)
> > @@ -348,9 +348,11 @@
> > } catch (KeeperException.NoNodeException e) {
> > return ensureParentExists(znode) && ensureExists(znode);
> > } catch (KeeperException e) {
> > - LOG.warn("Failed to create " + znode + ":", e);
> > + LOG.warn("Failed to create " + znode +
> > + " -- check quorum servers, currenty=" + this.quorumServers, e);
> > } catch (InterruptedException e) {
> > - LOG.warn("Failed to create " + znode + ":", e);
> > + LOG.warn("Failed to create " + znode +
> > + " -- check quorum servers, currenty=" + this.quorumServers, e);
> > }
> > return false;
> > }
> >
> > Looks like this in logs:
> >
> > 2009-07-24 14:48:40,923 WARN
> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create
> /hbase
> > -- check quorum servers, currenty=localhost:2181
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase
> >
> > St.Ack
> >
> >
> > On Fri, Jul 24, 2009 at 2:18 PM, Ken Weiner <ke...@gumgum.com> wrote:
> >
> > > After spending some more time looking at this, I HBase's current
> logging
> > is
> > > sufficient to indicate it is a ZK problem. Once switching the log level
> > to
> > > WARN, it prints:
> > >
> > > WARN (o.a.h.h.zookeeper.ZooKeeperWrapper) - Failed to create /hbase:
> > > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > > KeeperErrorCode = ConnectionLoss for /hbase
> > >
> > > Once improvement would be to log a message saying to check the ZK
> quorum
> > > server configuration and list what the current list of servers is, but
> > I'm
> > > not sure where in the code that should go.
> > >
> > > On Fri, Jul 24, 2009 at 11:42 AM, stack <st...@duboce.net> wrote:
> > >
> > > > Ken:
> > > >
> > > > For sure...make an issue and put up a patch. Looks like a little
> debug
> > > log
> > > > in there wouldn't go amiss at all.
> > > >
> > > > Thanks for figuring this lads,
> > > > St.Ack
> > > >
> > > >
> > >
> >
>
Re: Latest trunk - client throwing org.apache.hadoop.hbase.client.NoServerForRegionException:
Timed out trying to locate root region
Posted by Ken Weiner <ke...@gumgum.com>.
St.Ack,
That message looks great. I bet it will help a lot of people who
misconfigure things.
One minor spelling correction: "currenty=localhost:2181" should be
"currently=localhost:2181"
-Ken
On Fri, Jul 24, 2009 at 2:49 PM, stack <st...@duboce.net> wrote:
> Hows this:
>
> Index: src/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWrapper.java
> ===================================================================
> --- src/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWrapper.java
> (revision 797665)
> +++ src/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWrapper.java
> (working copy)
> @@ -348,9 +348,11 @@
> } catch (KeeperException.NoNodeException e) {
> return ensureParentExists(znode) && ensureExists(znode);
> } catch (KeeperException e) {
> - LOG.warn("Failed to create " + znode + ":", e);
> + LOG.warn("Failed to create " + znode +
> + " -- check quorum servers, currenty=" + this.quorumServers, e);
> } catch (InterruptedException e) {
> - LOG.warn("Failed to create " + znode + ":", e);
> + LOG.warn("Failed to create " + znode +
> + " -- check quorum servers, currenty=" + this.quorumServers, e);
> }
> return false;
> }
>
> Looks like this in logs:
>
> 2009-07-24 14:48:40,923 WARN
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create /hbase
> -- check quorum servers, currenty=localhost:2181
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
>
> St.Ack
>
>
> On Fri, Jul 24, 2009 at 2:18 PM, Ken Weiner <ke...@gumgum.com> wrote:
>
> > After spending some more time looking at this, I HBase's current logging
> is
> > sufficient to indicate it is a ZK problem. Once switching the log level
> to
> > WARN, it prints:
> >
> > WARN (o.a.h.h.zookeeper.ZooKeeperWrapper) - Failed to create /hbase:
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase
> >
> > Once improvement would be to log a message saying to check the ZK quorum
> > server configuration and list what the current list of servers is, but
> I'm
> > not sure where in the code that should go.
> >
> > On Fri, Jul 24, 2009 at 11:42 AM, stack <st...@duboce.net> wrote:
> >
> > > Ken:
> > >
> > > For sure...make an issue and put up a patch. Looks like a little debug
> > log
> > > in there wouldn't go amiss at all.
> > >
> > > Thanks for figuring this lads,
> > > St.Ack
> > >
> > >
> >
>
Re: Latest trunk - client throwing org.apache.hadoop.hbase.client.NoServerForRegionException:
Timed out trying to locate root region
Posted by stack <st...@duboce.net>.
Hows this:
Index: src/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWrapper.java
===================================================================
--- src/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWrapper.java
(revision 797665)
+++ src/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWrapper.java
(working copy)
@@ -348,9 +348,11 @@
} catch (KeeperException.NoNodeException e) {
return ensureParentExists(znode) && ensureExists(znode);
} catch (KeeperException e) {
- LOG.warn("Failed to create " + znode + ":", e);
+ LOG.warn("Failed to create " + znode +
+ " -- check quorum servers, currenty=" + this.quorumServers, e);
} catch (InterruptedException e) {
- LOG.warn("Failed to create " + znode + ":", e);
+ LOG.warn("Failed to create " + znode +
+ " -- check quorum servers, currenty=" + this.quorumServers, e);
}
return false;
}
Looks like this in logs:
2009-07-24 14:48:40,923 WARN
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create /hbase
-- check quorum servers, currenty=localhost:2181
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
St.Ack
On Fri, Jul 24, 2009 at 2:18 PM, Ken Weiner <ke...@gumgum.com> wrote:
> After spending some more time looking at this, I HBase's current logging is
> sufficient to indicate it is a ZK problem. Once switching the log level to
> WARN, it prints:
>
> WARN (o.a.h.h.zookeeper.ZooKeeperWrapper) - Failed to create /hbase:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
>
> Once improvement would be to log a message saying to check the ZK quorum
> server configuration and list what the current list of servers is, but I'm
> not sure where in the code that should go.
>
> On Fri, Jul 24, 2009 at 11:42 AM, stack <st...@duboce.net> wrote:
>
> > Ken:
> >
> > For sure...make an issue and put up a patch. Looks like a little debug
> log
> > in there wouldn't go amiss at all.
> >
> > Thanks for figuring this lads,
> > St.Ack
> >
> >
>
Re: Latest trunk - client throwing org.apache.hadoop.hbase.client.NoServerForRegionException:
Timed out trying to locate root region
Posted by Ken Weiner <ke...@gumgum.com>.
After spending some more time looking at this, I HBase's current logging is
sufficient to indicate it is a ZK problem. Once switching the log level to
WARN, it prints:
WARN (o.a.h.h.zookeeper.ZooKeeperWrapper) - Failed to create /hbase:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
Once improvement would be to log a message saying to check the ZK quorum
server configuration and list what the current list of servers is, but I'm
not sure where in the code that should go.
On Fri, Jul 24, 2009 at 11:42 AM, stack <st...@duboce.net> wrote:
> Ken:
>
> For sure...make an issue and put up a patch. Looks like a little debug log
> in there wouldn't go amiss at all.
>
> Thanks for figuring this lads,
> St.Ack
>
>
Re: Latest trunk - client throwing org.apache.hadoop.hbase.client.NoServerForRegionException:
Timed out trying to locate root region
Posted by stack <st...@duboce.net>.
Ken:
For sure...make an issue and put up a patch. Looks like a little debug log
in there wouldn't go amiss at all.
Thanks for figuring this lads,
St.Ack
On Fri, Jul 24, 2009 at 11:29 AM, Ken Weiner <ke...@gumgum.com> wrote:
> I work with Vaibhav. Being new to HBase/ZK integration, it took us a
> while
> to figure this out. I wonder if a message can be logged in
> ZookeeperWrapper
> when this happens so we know it has to do with ZK:
>
> private HServerAddress readAddress(String znode, Watcher watcher) {
> try {
> return readAddressOrThrow(znode, watcher);
> } catch (IOException e) {
> // Add a WARNING message here saying "not able to reach Zookeeper at
> znode" + znode
> return null;
> }
> }
>
> On Fri, Jul 24, 2009 at 11:26 AM, Vaibhav Puranik <vpuranik@gmail.com
> >wrote:
>
> > Hi all,
> >
> > Please ignore this message. This exception was the result of me putting
> > wrong hbase.zookeeper.quorum value in my client code.
> >
> > We are testing the latest truck and so far it's going very good!
> >
> > Regards,
> > Vaibhav
> >
> > On Fri, Jul 24, 2009 at 10:48 AM, Vaibhav Puranik <vpuranik@gmail.com
> > >wrote:
> >
> > > Hi,
> > >
> > > We have deployed the latest trunk (yesterday evening 5 pm) on EC2
> > >
> > > 1 master, 3 region servers.
> > >
> > > This morning, when I am trying to run client script for querying the
> > table,
> > > I am getting following exception:
> > >
> > > Caught: org.apache.hadoop.hbase.client.NoServerForRegionException:
> Timed
> > > out trying to locate root region
> > >
> > > Any ideas, suggestions?
> > >
> > > Regards,
> > > Vaibhav Puranik
> > > Gumgum
> > >
> >
>
Re: Latest trunk - client throwing org.apache.hadoop.hbase.client.NoServerForRegionException:
Timed out trying to locate root region
Posted by Ken Weiner <ke...@gumgum.com>.
I work with Vaibhav. Being new to HBase/ZK integration, it took us a while
to figure this out. I wonder if a message can be logged in ZookeeperWrapper
when this happens so we know it has to do with ZK:
private HServerAddress readAddress(String znode, Watcher watcher) {
try {
return readAddressOrThrow(znode, watcher);
} catch (IOException e) {
// Add a WARNING message here saying "not able to reach Zookeeper at
znode" + znode
return null;
}
}
On Fri, Jul 24, 2009 at 11:26 AM, Vaibhav Puranik <vp...@gmail.com>wrote:
> Hi all,
>
> Please ignore this message. This exception was the result of me putting
> wrong hbase.zookeeper.quorum value in my client code.
>
> We are testing the latest truck and so far it's going very good!
>
> Regards,
> Vaibhav
>
> On Fri, Jul 24, 2009 at 10:48 AM, Vaibhav Puranik <vpuranik@gmail.com
> >wrote:
>
> > Hi,
> >
> > We have deployed the latest trunk (yesterday evening 5 pm) on EC2
> >
> > 1 master, 3 region servers.
> >
> > This morning, when I am trying to run client script for querying the
> table,
> > I am getting following exception:
> >
> > Caught: org.apache.hadoop.hbase.client.NoServerForRegionException: Timed
> > out trying to locate root region
> >
> > Any ideas, suggestions?
> >
> > Regards,
> > Vaibhav Puranik
> > Gumgum
> >
>
Re: Latest trunk - client throwing org.apache.hadoop.hbase.client.NoServerForRegionException:
Timed out trying to locate root region
Posted by Vaibhav Puranik <vp...@gmail.com>.
Hi all,
Please ignore this message. This exception was the result of me putting
wrong hbase.zookeeper.quorum value in my client code.
We are testing the latest truck and so far it's going very good!
Regards,
Vaibhav
On Fri, Jul 24, 2009 at 10:48 AM, Vaibhav Puranik <vp...@gmail.com>wrote:
> Hi,
>
> We have deployed the latest trunk (yesterday evening 5 pm) on EC2
>
> 1 master, 3 region servers.
>
> This morning, when I am trying to run client script for querying the table,
> I am getting following exception:
>
> Caught: org.apache.hadoop.hbase.client.NoServerForRegionException: Timed
> out trying to locate root region
>
> Any ideas, suggestions?
>
> Regards,
> Vaibhav Puranik
> Gumgum
>