You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "shanmuganathan.r" <sh...@zohocorp.com> on 2011/08/02 13:12:14 UTC
Backup Master not started?
Hi All,
I have the problem in my HBase fully distributed mode with backup master . The HBase in the fully distributed mode is starting and working properly. The Active Master is working and My backup master is waiting for the active master termination. If I kill the active master process, the backup master don't come to active which remains in the waiting state.
I declared the backup master in the HBase/conf/Backup-masters file.
My backup master log is
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: revision
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUser
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsDate
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUrl
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: date
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsRevision
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: user
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsVersion
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: url
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: version
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
2011-08-02 16:04:16,417 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
2011-08-02 16:04:16,417 INFO org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
2011-08-02 16:04:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: HMaster started in backup mode. Stalling until master znode is written.
2011-08-02 16:04:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
2011-08-02 16:07:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
2011-08-02 16:10:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
2011-08-02 16:13:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
And the one of my region server log is
2011-08-02 16:21:45,977 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=958.41 KB, free=198.43 MB, max=199.36 MB, blocks=1, accesses=10, hits=9, hitRatio=89.99%%, cachingAccesses=10, cachingHits=9, cachingHitsRatio=89.99%%, evictions=0, evicted=0, evictedPerRun=NaN
2011-08-02 16:22:40,714 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy5.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1445)
at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:737)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:586)
at java.lang.Thread.run(Thread.java:636)
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1. How can we solve this problem?
2. Is there any extra configuration needed for this backup master configuration in HBase?
Thanks for your valuable suggestions ......!
Regards,
Shanmuganathan
Backup Master is not started?
Posted by "shanmuganathan.r" <sh...@zohocorp.com>.
Hi All,
I have the problem in my HBase fully distributed mode with backup master . The HBase in the fully distributed mode is starting and working properly. The Active Master is working and My backup master is waiting for the active master termination. If I kill the active master process, the backup master don't come to active which remains in the waiting state.
I declared the backup master in the HBase/conf/Backup-masters file.
My backup master log is
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: revision
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUser
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsDate
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUrl
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: date
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsRevision
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: user
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsVersion
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: url
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: version
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
2011-08-02 16:04:16,417 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
2011-08-02 16:04:16,417 INFO org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
2011-08-02 16:04:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: HMaster started in backup mode. Stalling until master znode is written.
2011-08-02 16:04:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
2011-08-02 16:07:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
2011-08-02 16:10:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
2011-08-02 16:13:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
And the one of my region server log is
2011-08-02 16:21:45,977 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=958.41 KB, free=198.43 MB, max=199.36 MB, blocks=1, accesses=10, hits=9, hitRatio=89.99%%, cachingAccesses=10, cachingHits=9, cachingHitsRatio=89.99%%, evictions=0, evicted=0, evictedPerRun=NaN
2011-08-02 16:22:40,714 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy5.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1445)
at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:737)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:586)
at java.lang.Thread.run(Thread.java:636)
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1. How can we solve this problem?
2. Is there any extra configuration needed for this backup master configuration in HBase?
Thanks for your valuable suggestions ......!
Regards,
Shanmuganathan
Re: Backup Master is not started?
Posted by "shanmuganathan.r" <sh...@zohocorp.com>.
Hi Jean,
Thank you for your suggestions.................
I am using the same configuration for the both active and backup masters. But the problem is not solved. The backup master is not take the master role. How can I solve this one?
Regards,
Shanmuganathan
---- On Wed, 03 Aug 2011 23:35:38 +0530 Jean-Daniel Cryans<jdcryans@apache.org> wrote ----
Have them use the same configuration?
J-D
On Wed, Aug 3, 2011 at 7:17 AM, shanmuganathan.r
<shanmuganathan.r@zohocorp.com> wrote:
> Hi Jean,
>
> The Problem is continued even the Active master is properly killed. How can we make the Active master and backup master use the same zookeeper ensemble.
>
> Regards,
>
> Shanmuganathan
>
>
>
> ---- On Tue, 02 Aug 2011 22:42:10 +0530 Jean-Daniel Cryans&lt;jdcryans@apache.org&gt; wrote ----
>
>
> Like it says, the backup master is waiting for the primary one to
> create the znode as in it considers it's not even running yet. If your
> primary node is indeed running correctly, it could mean that they
> aren't using the same Zookeeper ensemble or root znode.
>
> Check that first.
>
> J-D
>
> On Tue, Aug 2, 2011 at 4:12 AM, shanmuganathan.r
> &lt;shanmuganathan.r@zohocorp.com&gt; wrote:
> &gt; Hi All,
> &gt;
> &gt; I have the problem in my HBase fully distributed mode with backup master . The HBase in the fully distributed mode is starting and working properly. The Active Master is working and My backup master is waiting for the active master termination. If I kill the active master process, the backup master don't come to active which remains in the waiting state.
> &gt;
> &gt;
> &gt; I declared the backup master in the HBase/conf/Backup-masters file.
> &gt;
> &gt;
> &gt; My backup master log is
> &gt; 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: revision
> &gt; 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUser
> &gt; 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsDate
> &gt; 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUrl
> &gt; 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: date
> &gt; 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsRevision
> &gt; 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: user
> &gt; 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsVersion
> &gt; 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: url
> &gt; 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: version
> &gt; 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
> &gt; 2011-08-02 16:04:16,417 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
> &gt; 2011-08-02 16:04:16,417 INFO org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
> &gt; 2011-08-02 16:04:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: HMaster started in backup mode. Stalling until master znode is written.
> &gt; 2011-08-02 16:04:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
> &gt; 2011-08-02 16:07:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
> &gt; 2011-08-02 16:10:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
> &gt; 2011-08-02 16:13:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
> &gt;
> &gt;
> &gt; --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> &gt;
> &gt;
> &gt; And the one of my region server log is
> &gt;
> &gt;
> &gt; 2011-08-02 16:21:45,977 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=958.41 KB, free=198.43 MB, max=199.36 MB, blocks=1, accesses=10, hits=9, hitRatio=89.99%%, cachingAccesses=10, cachingHits=9, cachingHitsRatio=89.99%%, evictions=0, evicted=0, evictedPerRun=NaN
> &gt; 2011-08-02 16:22:40,714 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was:
> &gt; java.net.ConnectException: Connection refused
> &gt; at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> &gt; at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> &gt; at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> &gt; at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> &gt; at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
> &gt; at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
> &gt; at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
> &gt; at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> &gt; at $Proxy5.getProtocolVersion(Unknown Source)
> &gt; at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
> &gt; at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
> &gt; at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
> &gt; at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
> &gt; at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1445)
> &gt; at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:737)
> &gt; at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:586)
> &gt; at java.lang.Thread.run(Thread.java:636)
> &gt;
> &gt;
> &gt;
> &gt; --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> &gt;
> &gt;
> &gt; 1. How can we solve this problem?
> &gt;
> &gt;
> &gt; 2. Is there any extra configuration needed for this backup master configuration in HBase?
> &gt;
> &gt;
> &gt; Thanks for your valuable suggestions ......!
> &gt;
> &gt;
> &gt; Regards,
> &gt;
> &gt; Shanmuganathan
> &gt;
> &gt;
> &gt;
> &gt;
>
>
>
Re: Backup Master not started?
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Have them use the same configuration?
J-D
On Wed, Aug 3, 2011 at 7:17 AM, shanmuganathan.r
<sh...@zohocorp.com> wrote:
> Hi Jean,
>
> The Problem is continued even the Active master is properly killed. How can we make the Active master and backup master use the same zookeeper ensemble.
>
> Regards,
>
> Shanmuganathan
>
>
>
> ---- On Tue, 02 Aug 2011 22:42:10 +0530 Jean-Daniel Cryans<jdcryans@apache.org> wrote ----
>
>
> Like it says, the backup master is waiting for the primary one to
> create the znode as in it considers it's not even running yet. If your
> primary node is indeed running correctly, it could mean that they
> aren't using the same Zookeeper ensemble or root znode.
>
> Check that first.
>
> J-D
>
> On Tue, Aug 2, 2011 at 4:12 AM, shanmuganathan.r
> <shanmuganathan.r@zohocorp.com> wrote:
> > Hi All,
> >
> > I have the problem in my HBase fully distributed mode with backup master . The HBase in the fully distributed mode is starting and working properly. The Active Master is working and My backup master is waiting for the active master termination. If I kill the active master process, the backup master don't come to active which remains in the waiting state.
> >
> >
> > I declared the backup master in the HBase/conf/Backup-masters file.
> >
> >
> > My backup master log is
> > 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: revision
> > 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUser
> > 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsDate
> > 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUrl
> > 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: date
> > 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsRevision
> > 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: user
> > 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsVersion
> > 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: url
> > 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: version
> > 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
> > 2011-08-02 16:04:16,417 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
> > 2011-08-02 16:04:16,417 INFO org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
> > 2011-08-02 16:04:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: HMaster started in backup mode. Stalling until master znode is written.
> > 2011-08-02 16:04:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
> > 2011-08-02 16:07:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
> > 2011-08-02 16:10:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
> > 2011-08-02 16:13:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
> >
> >
> > --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >
> >
> > And the one of my region server log is
> >
> >
> > 2011-08-02 16:21:45,977 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=958.41 KB, free=198.43 MB, max=199.36 MB, blocks=1, accesses=10, hits=9, hitRatio=89.99%%, cachingAccesses=10, cachingHits=9, cachingHitsRatio=89.99%%, evictions=0, evicted=0, evictedPerRun=NaN
> > 2011-08-02 16:22:40,714 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was:
> > java.net.ConnectException: Connection refused
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> > at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> > at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
> > at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
> > at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> > at $Proxy5.getProtocolVersion(Unknown Source)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
> > at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1445)
> > at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:737)
> > at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:586)
> > at java.lang.Thread.run(Thread.java:636)
> >
> >
> >
> > --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >
> >
> > 1. How can we solve this problem?
> >
> >
> > 2. Is there any extra configuration needed for this backup master configuration in HBase?
> >
> >
> > Thanks for your valuable suggestions ......!
> >
> >
> > Regards,
> >
> > Shanmuganathan
> >
> >
> >
> >
>
>
>
Re: Backup Master not started?
Posted by "shanmuganathan.r" <sh...@zohocorp.com>.
Hi Jean,
The Problem is continued even the Active master is properly killed. How can we make the Active master and backup master use the same zookeeper ensemble.
Regards,
Shanmuganathan
---- On Tue, 02 Aug 2011 22:42:10 +0530 Jean-Daniel Cryans<jdcryans@apache.org> wrote ----
Like it says, the backup master is waiting for the primary one to
create the znode as in it considers it's not even running yet. If your
primary node is indeed running correctly, it could mean that they
aren't using the same Zookeeper ensemble or root znode.
Check that first.
J-D
On Tue, Aug 2, 2011 at 4:12 AM, shanmuganathan.r
<shanmuganathan.r@zohocorp.com> wrote:
> Hi All,
>
> I have the problem in my HBase fully distributed mode with backup master . The HBase in the fully distributed mode is starting and working properly. The Active Master is working and My backup master is waiting for the active master termination. If I kill the active master process, the backup master don't come to active which remains in the waiting state.
>
>
> I declared the backup master in the HBase/conf/Backup-masters file.
>
>
> My backup master log is
> 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: revision
> 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUser
> 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsDate
> 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUrl
> 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: date
> 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsRevision
> 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: user
> 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsVersion
> 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: url
> 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: version
> 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
> 2011-08-02 16:04:16,417 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
> 2011-08-02 16:04:16,417 INFO org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
> 2011-08-02 16:04:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: HMaster started in backup mode. Stalling until master znode is written.
> 2011-08-02 16:04:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
> 2011-08-02 16:07:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
> 2011-08-02 16:10:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
> 2011-08-02 16:13:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
>
>
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>
> And the one of my region server log is
>
>
> 2011-08-02 16:21:45,977 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=958.41 KB, free=198.43 MB, max=199.36 MB, blocks=1, accesses=10, hits=9, hitRatio=89.99%%, cachingAccesses=10, cachingHits=9, cachingHitsRatio=89.99%%, evictions=0, evicted=0, evictedPerRun=NaN
> 2011-08-02 16:22:40,714 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was:
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
> at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> at $Proxy5.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1445)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:737)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:586)
> at java.lang.Thread.run(Thread.java:636)
>
>
>
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>
> 1. How can we solve this problem?
>
>
> 2. Is there any extra configuration needed for this backup master configuration in HBase?
>
>
> Thanks for your valuable suggestions ......!
>
>
> Regards,
>
> Shanmuganathan
>
>
>
>
Re: Backup Master not started?
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Like it says, the backup master is waiting for the primary one to
create the znode as in it considers it's not even running yet. If your
primary node is indeed running correctly, it could mean that they
aren't using the same Zookeeper ensemble or root znode.
Check that first.
J-D
On Tue, Aug 2, 2011 at 4:12 AM, shanmuganathan.r
<sh...@zohocorp.com> wrote:
> Hi All,
>
> I have the problem in my HBase fully distributed mode with backup master . The HBase in the fully distributed mode is starting and working properly. The Active Master is working and My backup master is waiting for the active master termination. If I kill the active master process, the backup master don't come to active which remains in the waiting state.
>
>
> I declared the backup master in the HBase/conf/Backup-masters file.
>
>
> My backup master log is
> 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: revision
> 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUser
> 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsDate
> 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUrl
> 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: date
> 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsRevision
> 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: user
> 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsVersion
> 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: url
> 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: version
> 2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
> 2011-08-02 16:04:16,417 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
> 2011-08-02 16:04:16,417 INFO org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
> 2011-08-02 16:04:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: HMaster started in backup mode. Stalling until master znode is written.
> 2011-08-02 16:04:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
> 2011-08-02 16:07:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
> 2011-08-02 16:10:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
> 2011-08-02 16:13:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
>
>
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>
> And the one of my region server log is
>
>
> 2011-08-02 16:21:45,977 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=958.41 KB, free=198.43 MB, max=199.36 MB, blocks=1, accesses=10, hits=9, hitRatio=89.99%%, cachingAccesses=10, cachingHits=9, cachingHitsRatio=89.99%%, evictions=0, evicted=0, evictedPerRun=NaN
> 2011-08-02 16:22:40,714 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was:
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
> at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> at $Proxy5.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1445)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:737)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:586)
> at java.lang.Thread.run(Thread.java:636)
>
>
>
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>
> 1. How can we solve this problem?
>
>
> 2. Is there any extra configuration needed for this backup master configuration in HBase?
>
>
> Thanks for your valuable suggestions ......!
>
>
> Regards,
>
> Shanmuganathan
>
>
>
>
Backup Master is not started?
Posted by "shanmuganathan.r" <sh...@zohocorp.com>.
Hi All,
I have the problem in my HBase fully distributed mode with backup master . The HBase in the fully distributed mode is starting and working properly. The Active Master is working and My backup master is waiting for the active master termination. If I kill the active master process, the backup master don't come to active which remains in the waiting state.
I declared the backup master in the HBase/conf/Backup-masters file.
My backup master log is
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: revision
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUser
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsDate
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUrl
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: date
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsRevision
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: user
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsVersion
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: url
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: MetricsString added: version
2011-08-02 16:04:16,416 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
2011-08-02 16:04:16,417 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
2011-08-02 16:04:16,417 INFO org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
2011-08-02 16:04:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: HMaster started in backup mode. Stalling until master znode is written.
2011-08-02 16:04:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
2011-08-02 16:07:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
2011-08-02 16:10:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
2011-08-02 16:13:16,418 DEBUG org.apache.hadoop.hbase.master.HMaster: Waiting for master address ZNode to be written (Also watching cluster state node)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
And the one of my region server log is
2011-08-02 16:21:45,977 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=958.41 KB, free=198.43 MB, max=199.36 MB, blocks=1, accesses=10, hits=9, hitRatio=89.99%%, cachingAccesses=10, cachingHits=9, cachingHitsRatio=89.99%%, evictions=0, evicted=0, evictedPerRun=NaN
2011-08-02 16:22:40,714 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy5.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1445)
at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:737)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:586)
at java.lang.Thread.run(Thread.java:636)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1. How can we solve this problem?
2. Is there any extra configuration needed for this backup master configuration in HBase?
Thanks for your valuable suggestions ......!
Regards,
Shanmuganathan