You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Brian Jeltema <br...@digitalenvoy.net> on 2013/07/09 19:05:00 UTC
ClusterId read in ZooKeeper is null
I'm new to HBase, and need a little guidance. I've set up a 6-node cluster, with 3 nodes
running the ZooKeeper server. The database seems to be working from the hbase shell; I can create tables, insert,
scan, etc.
But when I try to perform operations in a Java app, I hang at:
13/07/09 12:40:34 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=cas-2:2181,cas-1:2181,cas-3:2181 sessionTimeout=90000 watcher=hconnection-0x6833f0de
13/07/09 12:40:34 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x6833f0de connecting to ZooKeeper ensemble=cas-2:2181,cas-1:2181,cas-3:2181
13/07/09 12:40:34 INFO zookeeper.ClientCnxn: Opening socket connection to server cas-1/10.4.0.1:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration)
13/07/09 12:40:34 INFO zookeeper.ClientCnxn: Socket connection established to cas-1/10.4.0.1:2181, initiating session
13/07/09 12:40:34 INFO zookeeper.ClientCnxn: Session establishment complete on server cas-1/10.4.0.1:2181, sessionid = 0x13fa5b5096e001f, negotiated timeout = 40000
13/07/09 12:40:34 INFO client.ZooKeeperRegistry: ClusterId read in ZooKeeper is null
The Java code is nothing more than:
Configuration hConf = HBaseConfiguration.create();
hConf.set("hbase.zookeeper.quorum", "cas-1,cas-2,cas-3");
hConf.set("hbase.zookeeper.property.clientPort", "2181");
HTable hTable = new HTable(hConf, "tablename");
a thread dump shows the app blocked:
"main" prio=10 tid=0x00007f1424009000 nid=0x2976 waiting on condition [0x00007f142a0c3000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.blockUntilAvailable(ZKUtil.java:1772)
at org.apache.hadoop.hbase.zookeeper.MetaRegionTracker.blockUntilAvailable(MetaRegionTracker.java:175)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:58)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:806)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:896)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:809)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:778)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:245)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:186)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:147)
Any suggestions as to the cause?
TIA
Brian
Re: ClusterId read in ZooKeeper is null
Posted by Brian Jeltema <br...@digitalenvoy.net>.
This issue has been resolved. It was caused by version skew between the client library and the running service.
On Jul 10, 2013, at 11:47 AM, Brian Jeltema wrote:
> As far as I can tell the HMaster process is running correctly. There are no obvious problems in the logs.
> As suggested, I defined zookeeper.znode.parent in the client to match the deployment, which did eliminate
> the warning about the null ClusterId. However, the client still hangs in the same place.
>
> After some debugging, the client appears to be looking for a zookeeper node named /hbase_unsecure/meta-region-server.
> From the zookeeper cli (slightly edited):
>
> [zk: ] ls /hbase-unsecure
>
> returns
>
> [splitlog, online-snapshot, unassigned, root-region-server, rs, table92, backup-masters, draining, table, master, shutdown, hbaseid]
>
> so meta-region-server node does not exist. As I read the code, the HBase Master would have set that in a postOpenDeployTasks method, which
> apparently is not being called. At this point it's not clear to me what else to look for.
>
> Brian
>
> On Jul 9, 2013, at 1:28 PM, Gary Helmling wrote:
>
>> Is the HMaster process running correctly on the cluster? Between the
>> missing cluster ID and meta region not being available, it looks like
>> HMaster may not have fully initialized.
>>
>> Alternately, if HMaster is running correctly, did you override the default
>> value for zookeeper.znode.parent in your cluster configuration, but not set
>> this in your client code?
>>
>>
>> On Tue, Jul 9, 2013 at 10:05 AM, Brian Jeltema <
>> brian.jeltema@digitalenvoy.net> wrote:
>>
>>> I'm new to HBase, and need a little guidance. I've set up a 6-node
>>> cluster, with 3 nodes
>>> running the ZooKeeper server. The database seems to be working from the
>>> hbase shell; I can create tables, insert,
>>> scan, etc.
>>>
>>> But when I try to perform operations in a Java app, I hang at:
>>>
>>> 13/07/09 12:40:34 INFO zookeeper.ZooKeeper: Initiating client connection,
>>> connectString=cas-2:2181,cas-1:2181,cas-3:2181 sessionTimeout=90000
>>> watcher=hconnection-0x6833f0de
>>> 13/07/09 12:40:34 INFO zookeeper.RecoverableZooKeeper: Process
>>> identifier=hconnection-0x6833f0de connecting to ZooKeeper
>>> ensemble=cas-2:2181,cas-1:2181,cas-3:2181
>>> 13/07/09 12:40:34 INFO zookeeper.ClientCnxn: Opening socket connection to
>>> server cas-1/10.4.0.1:2181. Will not attempt to authenticate using SASL
>>> (Unable to locate a login configuration)
>>> 13/07/09 12:40:34 INFO zookeeper.ClientCnxn: Socket connection established
>>> to cas-1/10.4.0.1:2181, initiating session
>>> 13/07/09 12:40:34 INFO zookeeper.ClientCnxn: Session establishment
>>> complete on server cas-1/10.4.0.1:2181, sessionid = 0x13fa5b5096e001f,
>>> negotiated timeout = 40000
>>> 13/07/09 12:40:34 INFO client.ZooKeeperRegistry: ClusterId read in
>>> ZooKeeper is null
>>>
>>> The Java code is nothing more than:
>>>
>>> Configuration hConf = HBaseConfiguration.create();
>>> hConf.set("hbase.zookeeper.quorum", "cas-1,cas-2,cas-3");
>>> hConf.set("hbase.zookeeper.property.clientPort", "2181");
>>> HTable hTable = new HTable(hConf, "tablename");
>>>
>>> a thread dump shows the app blocked:
>>>
>>> "main" prio=10 tid=0x00007f1424009000 nid=0x2976 waiting on condition
>>> [0x00007f142a0c3000]
>>> java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> at java.lang.Thread.sleep(Native Method)
>>> at
>>> org.apache.hadoop.hbase.zookeeper.ZKUtil.blockUntilAvailable(ZKUtil.java:1772)
>>> at
>>> org.apache.hadoop.hbase.zookeeper.MetaRegionTracker.blockUntilAvailable(MetaRegionTracker.java:175)
>>> at
>>> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:58)
>>> at
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:806)
>>> at
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:896)
>>> at
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:809)
>>> at
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:778)
>>> at
>>> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:245)
>>> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:186)
>>> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:147)
>>>
>>> Any suggestions as to the cause?
>>>
>>> TIA
>>>
>>> Brian
>
>
Re: ClusterId read in ZooKeeper is null
Posted by Brian Jeltema <br...@digitalenvoy.net>.
As far as I can tell the HMaster process is running correctly. There are no obvious problems in the logs.
As suggested, I defined zookeeper.znode.parent in the client to match the deployment, which did eliminate
the warning about the null ClusterId. However, the client still hangs in the same place.
After some debugging, the client appears to be looking for a zookeeper node named /hbase_unsecure/meta-region-server.
From the zookeeper cli (slightly edited):
[zk: ] ls /hbase-unsecure
returns
[splitlog, online-snapshot, unassigned, root-region-server, rs, table92, backup-masters, draining, table, master, shutdown, hbaseid]
so meta-region-server node does not exist. As I read the code, the HBase Master would have set that in a postOpenDeployTasks method, which
apparently is not being called. At this point it's not clear to me what else to look for.
Brian
On Jul 9, 2013, at 1:28 PM, Gary Helmling wrote:
> Is the HMaster process running correctly on the cluster? Between the
> missing cluster ID and meta region not being available, it looks like
> HMaster may not have fully initialized.
>
> Alternately, if HMaster is running correctly, did you override the default
> value for zookeeper.znode.parent in your cluster configuration, but not set
> this in your client code?
>
>
> On Tue, Jul 9, 2013 at 10:05 AM, Brian Jeltema <
> brian.jeltema@digitalenvoy.net> wrote:
>
>> I'm new to HBase, and need a little guidance. I've set up a 6-node
>> cluster, with 3 nodes
>> running the ZooKeeper server. The database seems to be working from the
>> hbase shell; I can create tables, insert,
>> scan, etc.
>>
>> But when I try to perform operations in a Java app, I hang at:
>>
>> 13/07/09 12:40:34 INFO zookeeper.ZooKeeper: Initiating client connection,
>> connectString=cas-2:2181,cas-1:2181,cas-3:2181 sessionTimeout=90000
>> watcher=hconnection-0x6833f0de
>> 13/07/09 12:40:34 INFO zookeeper.RecoverableZooKeeper: Process
>> identifier=hconnection-0x6833f0de connecting to ZooKeeper
>> ensemble=cas-2:2181,cas-1:2181,cas-3:2181
>> 13/07/09 12:40:34 INFO zookeeper.ClientCnxn: Opening socket connection to
>> server cas-1/10.4.0.1:2181. Will not attempt to authenticate using SASL
>> (Unable to locate a login configuration)
>> 13/07/09 12:40:34 INFO zookeeper.ClientCnxn: Socket connection established
>> to cas-1/10.4.0.1:2181, initiating session
>> 13/07/09 12:40:34 INFO zookeeper.ClientCnxn: Session establishment
>> complete on server cas-1/10.4.0.1:2181, sessionid = 0x13fa5b5096e001f,
>> negotiated timeout = 40000
>> 13/07/09 12:40:34 INFO client.ZooKeeperRegistry: ClusterId read in
>> ZooKeeper is null
>>
>> The Java code is nothing more than:
>>
>> Configuration hConf = HBaseConfiguration.create();
>> hConf.set("hbase.zookeeper.quorum", "cas-1,cas-2,cas-3");
>> hConf.set("hbase.zookeeper.property.clientPort", "2181");
>> HTable hTable = new HTable(hConf, "tablename");
>>
>> a thread dump shows the app blocked:
>>
>> "main" prio=10 tid=0x00007f1424009000 nid=0x2976 waiting on condition
>> [0x00007f142a0c3000]
>> java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at
>> org.apache.hadoop.hbase.zookeeper.ZKUtil.blockUntilAvailable(ZKUtil.java:1772)
>> at
>> org.apache.hadoop.hbase.zookeeper.MetaRegionTracker.blockUntilAvailable(MetaRegionTracker.java:175)
>> at
>> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:58)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:806)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:896)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:809)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:778)
>> at
>> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:245)
>> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:186)
>> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:147)
>>
>> Any suggestions as to the cause?
>>
>> TIA
>>
>> Brian
Re: ClusterId read in ZooKeeper is null
Posted by Gary Helmling <gh...@gmail.com>.
Is the HMaster process running correctly on the cluster? Between the
missing cluster ID and meta region not being available, it looks like
HMaster may not have fully initialized.
Alternately, if HMaster is running correctly, did you override the default
value for zookeeper.znode.parent in your cluster configuration, but not set
this in your client code?
On Tue, Jul 9, 2013 at 10:05 AM, Brian Jeltema <
brian.jeltema@digitalenvoy.net> wrote:
> I'm new to HBase, and need a little guidance. I've set up a 6-node
> cluster, with 3 nodes
> running the ZooKeeper server. The database seems to be working from the
> hbase shell; I can create tables, insert,
> scan, etc.
>
> But when I try to perform operations in a Java app, I hang at:
>
> 13/07/09 12:40:34 INFO zookeeper.ZooKeeper: Initiating client connection,
> connectString=cas-2:2181,cas-1:2181,cas-3:2181 sessionTimeout=90000
> watcher=hconnection-0x6833f0de
> 13/07/09 12:40:34 INFO zookeeper.RecoverableZooKeeper: Process
> identifier=hconnection-0x6833f0de connecting to ZooKeeper
> ensemble=cas-2:2181,cas-1:2181,cas-3:2181
> 13/07/09 12:40:34 INFO zookeeper.ClientCnxn: Opening socket connection to
> server cas-1/10.4.0.1:2181. Will not attempt to authenticate using SASL
> (Unable to locate a login configuration)
> 13/07/09 12:40:34 INFO zookeeper.ClientCnxn: Socket connection established
> to cas-1/10.4.0.1:2181, initiating session
> 13/07/09 12:40:34 INFO zookeeper.ClientCnxn: Session establishment
> complete on server cas-1/10.4.0.1:2181, sessionid = 0x13fa5b5096e001f,
> negotiated timeout = 40000
> 13/07/09 12:40:34 INFO client.ZooKeeperRegistry: ClusterId read in
> ZooKeeper is null
>
> The Java code is nothing more than:
>
> Configuration hConf = HBaseConfiguration.create();
> hConf.set("hbase.zookeeper.quorum", "cas-1,cas-2,cas-3");
> hConf.set("hbase.zookeeper.property.clientPort", "2181");
> HTable hTable = new HTable(hConf, "tablename");
>
> a thread dump shows the app blocked:
>
> "main" prio=10 tid=0x00007f1424009000 nid=0x2976 waiting on condition
> [0x00007f142a0c3000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.blockUntilAvailable(ZKUtil.java:1772)
> at
> org.apache.hadoop.hbase.zookeeper.MetaRegionTracker.blockUntilAvailable(MetaRegionTracker.java:175)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:58)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:806)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:896)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:809)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:778)
> at
> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:245)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:186)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:147)
>
> Any suggestions as to the cause?
>
> TIA
>
> Brian