You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by shivanandpawar <sh...@gmail.com> on 2015/03/04 08:02:40 UTC
Hbase Regionserver shutting down
We have installed a small Hadoop cluster with 3 machines.
We use the following code to connect to Hbase from java client.
Configuration config = HBaseConfiguration.create()
config.set(“hbase.zookeeper.quorum”, “xxxxxx”);
config.set(“hbase.zookeeper.property.clientPort”, “2181”);
config.set(“hbase.client.operation.timeout”, “1000”);
HConnection connection = HConnectionManager.createConnection(config);
HTableInterface hTable = connection.getTable("tableName");
But somehow the Hbase regionservers automatically shuts down after some time
giving the following error.
2015-02-23 03:38:49,472 WARN [regionserver60020]
zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper,
quorum=ip-10-174-18-85.ap-southeast-1.compute.internal:2181,ip-10-149-130-178.ap-southeast-1.compute.internal:2181,ip-10-174-28-180.ap-southeast-1.compute.internal:2181,
exception=org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for
/hbase/rs/ip-10-174-28-180.ap-southeast-1.compute.internal,60020,1424680062858
2015-02-23 03:38:49,472 ERROR [regionserver60020]
zookeeper.RecoverableZooKeeper: ZooKeeper delete failed after 4 attempts
2015-02-23 03:38:49,472 WARN [regionserver60020] regionserver.HRegionServer:
Failed deleting my ephemeral node
org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for
/hbase/rs/ip-10-174-28-180.ap-southeast-1.compute.internal,60020,1424680062858
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:156)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1270)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1259)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1286)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1016)
at java.lang.Thread.run(Thread.java:744)
2015-02-23 03:39:10,233 INFO [regionserver60020] regionserver.HRegionServer:
stopping server
ip-10-174-28-180.ap-southeast-1.compute.internal,60020,1424680062858;
zookeeper connection closed.
2015-02-23 03:39:10,233 INFO [regionserver60020] regionserver.HRegionServer:
regionserver60020 exiting
2015-02-23 03:39:10,236 ERROR [main] regionserver.HRegionServerCommandLine:
Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
at
org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:66)
at
org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:85)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2403)
2015-02-23 03:39:10,240 INFO [Thread-11] regionserver.ShutdownHook: Shutdown
hook starting; hbase.shutdown.hook=true;
fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@abde0e7
I tried increasing the JVM Heap size of regioserver to 16GB and session time
out to 120s. But nothing seems to help.
Please help to get the issue resolved??
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Hbase-Regionserver-shutting-down-tp4068834.html
Sent from the HBase Developer mailing list archive at Nabble.com.
Re: Hbase Regionserver shutting down
Posted by shivanandpawar <sh...@gmail.com>.
We are using HDP2.1. We removed the client application and after monitoring
for 2-3 days, the same problem occurred and the regionserver went down
giving the same error.
So we believe it is not the problem of client version mismatch.
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Hbase-Regionserver-shutting-down-tp4068834p4068863.html
Sent from the HBase Developer mailing list archive at Nabble.com.
Re: Hbase Regionserver shutting down
Posted by Vladimir Rodionov <vl...@gmail.com>.
>> We are using hbase version *0.98.0.2.1.7.0-784-hadoop2* and hbase client
>> version 0.96.0-hadoop2
Must be the same version for both.
If you are on 0.98 why is not the latest?
If you use cloudera/hortonworks distribution - provide the distro version.
On Wed, Mar 4, 2015 at 6:39 AM, shivanandpawar <sh...@gmail.com>
wrote:
> Hi,
>
> Thanks a lot for your reply.
>
> I have pasted the logs in pastebin and this is the link to that.
> http://pastebin.com/1Zn8Xtfx.
>
> We are using hbase version *0.98.0.2.1.7.0-784-hadoop2* and hbase client
> version 0.96.0-hadoop2.
>
>
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Hbase-Regionserver-shutting-down-tp4068834p4068845.html
> Sent from the HBase Developer mailing list archive at Nabble.com.
>
Re: Hbase Regionserver shutting down
Posted by shivanandpawar <sh...@gmail.com>.
Hi,
Thanks a lot for your reply.
I have pasted the logs in pastebin and this is the link to that.
http://pastebin.com/1Zn8Xtfx.
We are using hbase version *0.98.0.2.1.7.0-784-hadoop2* and hbase client
version 0.96.0-hadoop2.
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Hbase-Regionserver-shutting-down-tp4068834p4068845.html
Sent from the HBase Developer mailing list archive at Nabble.com.
Re: Hbase Regionserver shutting down
Posted by Esteban Gutierrez <es...@cloudera.com>.
-dev
+user
Hi,
Please try upload the logs from this RegionServer to pastebin.com or
gist.github.com so we can look into that. Also which version of HBase are
you using?
cheers,
esteban.
--
Cloudera, Inc.
On Tue, Mar 3, 2015 at 11:02 PM, shivanandpawar <sh...@gmail.com>
wrote:
> We have installed a small Hadoop cluster with 3 machines.
>
> We use the following code to connect to Hbase from java client.
> Configuration config = HBaseConfiguration.create()
> config.set(“hbase.zookeeper.quorum”, “xxxxxx”);
> config.set(“hbase.zookeeper.property.clientPort”, “2181”);
> config.set(“hbase.client.operation.timeout”, “1000”);
> HConnection connection = HConnectionManager.createConnection(config);
> HTableInterface hTable = connection.getTable("tableName");
>
> But somehow the Hbase regionservers automatically shuts down after some
> time
> giving the following error.
> 2015-02-23 03:38:49,472 WARN [regionserver60020]
> zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper,
>
> quorum=ip-10-174-18-85.ap-southeast-1.compute.internal:2181,ip-10-149-130-178.ap-southeast-1.compute.internal:2181,ip-10-174-28-180.ap-southeast-1.compute.internal:2181,
> exception=org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for
>
> /hbase/rs/ip-10-174-28-180.ap-southeast-1.compute.internal,60020,1424680062858
> 2015-02-23 03:38:49,472 ERROR [regionserver60020]
> zookeeper.RecoverableZooKeeper: ZooKeeper delete failed after 4 attempts
> 2015-02-23 03:38:49,472 WARN [regionserver60020]
> regionserver.HRegionServer:
> Failed deleting my ephemeral node
> org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for
>
> /hbase/rs/ip-10-174-28-180.ap-southeast-1.compute.internal,60020,1424680062858
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
> at
>
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:156)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1270)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1259)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1286)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1016)
> at java.lang.Thread.run(Thread.java:744)
> 2015-02-23 03:39:10,233 INFO [regionserver60020]
> regionserver.HRegionServer:
> stopping server
> ip-10-174-28-180.ap-southeast-1.compute.internal,60020,1424680062858;
> zookeeper connection closed.
> 2015-02-23 03:39:10,233 INFO [regionserver60020]
> regionserver.HRegionServer:
> regionserver60020 exiting
> 2015-02-23 03:39:10,236 ERROR [main] regionserver.HRegionServerCommandLine:
> Region server exiting
> java.lang.RuntimeException: HRegionServer Aborted
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:66)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:85)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
>
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2403)
> 2015-02-23 03:39:10,240 INFO [Thread-11] regionserver.ShutdownHook:
> Shutdown
> hook starting; hbase.shutdown.hook=true;
>
> fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@abde0e7
>
> I tried increasing the JVM Heap size of regioserver to 16GB and session
> time
> out to 120s. But nothing seems to help.
>
>
> Please help to get the issue resolved??
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Hbase-Regionserver-shutting-down-tp4068834.html
> Sent from the HBase Developer mailing list archive at Nabble.com.
>
Re: Hbase Regionserver shutting down
Posted by Esteban Gutierrez <es...@cloudera.com>.
-dev
+user
Hi,
Please try upload the logs from this RegionServer to pastebin.com or
gist.github.com so we can look into that. Also which version of HBase are
you using?
cheers,
esteban.
--
Cloudera, Inc.
On Tue, Mar 3, 2015 at 11:02 PM, shivanandpawar <sh...@gmail.com>
wrote:
> We have installed a small Hadoop cluster with 3 machines.
>
> We use the following code to connect to Hbase from java client.
> Configuration config = HBaseConfiguration.create()
> config.set(“hbase.zookeeper.quorum”, “xxxxxx”);
> config.set(“hbase.zookeeper.property.clientPort”, “2181”);
> config.set(“hbase.client.operation.timeout”, “1000”);
> HConnection connection = HConnectionManager.createConnection(config);
> HTableInterface hTable = connection.getTable("tableName");
>
> But somehow the Hbase regionservers automatically shuts down after some
> time
> giving the following error.
> 2015-02-23 03:38:49,472 WARN [regionserver60020]
> zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper,
>
> quorum=ip-10-174-18-85.ap-southeast-1.compute.internal:2181,ip-10-149-130-178.ap-southeast-1.compute.internal:2181,ip-10-174-28-180.ap-southeast-1.compute.internal:2181,
> exception=org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for
>
> /hbase/rs/ip-10-174-28-180.ap-southeast-1.compute.internal,60020,1424680062858
> 2015-02-23 03:38:49,472 ERROR [regionserver60020]
> zookeeper.RecoverableZooKeeper: ZooKeeper delete failed after 4 attempts
> 2015-02-23 03:38:49,472 WARN [regionserver60020]
> regionserver.HRegionServer:
> Failed deleting my ephemeral node
> org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for
>
> /hbase/rs/ip-10-174-28-180.ap-southeast-1.compute.internal,60020,1424680062858
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
> at
>
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:156)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1270)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1259)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1286)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1016)
> at java.lang.Thread.run(Thread.java:744)
> 2015-02-23 03:39:10,233 INFO [regionserver60020]
> regionserver.HRegionServer:
> stopping server
> ip-10-174-28-180.ap-southeast-1.compute.internal,60020,1424680062858;
> zookeeper connection closed.
> 2015-02-23 03:39:10,233 INFO [regionserver60020]
> regionserver.HRegionServer:
> regionserver60020 exiting
> 2015-02-23 03:39:10,236 ERROR [main] regionserver.HRegionServerCommandLine:
> Region server exiting
> java.lang.RuntimeException: HRegionServer Aborted
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:66)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:85)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
>
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2403)
> 2015-02-23 03:39:10,240 INFO [Thread-11] regionserver.ShutdownHook:
> Shutdown
> hook starting; hbase.shutdown.hook=true;
>
> fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@abde0e7
>
> I tried increasing the JVM Heap size of regioserver to 16GB and session
> time
> out to 120s. But nothing seems to help.
>
>
> Please help to get the issue resolved??
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Hbase-Regionserver-shutting-down-tp4068834.html
> Sent from the HBase Developer mailing list archive at Nabble.com.
>