答复: 答复: 答复: Region has been CLOSING for too long, this should eventually complete or the server will expire, send RPC again

[dylan <dw...@gmail.com>]

QJM zk configuration in core-site.xml

<property>

   <name>ha.zookeeper.quorum</name>

   <value>Slave01:2181,Slave02:2181,Slave03:2181</value>

</property>

 

<property>

      <name>hbase.zookeeper.quorum</name>

      <value>Slave01,Slave02,Slave03</value>

  </property>

 

hbase-home/logs have no zookeeper log. My zookeeper log located
zookeeper-home/data/version-2/

 

发件人: Azuryy Yu [mailto:azuryyyu@gmail.com] 
发送时间: 2013年4月16日 10:42
收件人: user@hadoop.apache.org
主题: Re: 答复: 答复: Region has been CLOSING for too long, this should
eventually complete or the server will expire, send RPC again

 

it located under hbase-home/logs/  if your zookeeper is managed by hbase.

 

but I noticed you configured QJM, then did your QJM and Hbase share the same
ZK cluster? if so, then just paste your QJM zk configuration in the
hdfs-site.xml and hbase zk configuration in the hbase-site.xml.

 

On Tue, Apr 16, 2013 at 10:37 AM, dylan <dw...@gmail.com> wrote:


How to check zookeeper log?? It is the binary files, how to transform it to
normal log? 


 


I find the “org.apache.zookeeper.server.LogFormatter”, how to run?


 


 

发件人: Azuryy Yu [mailto:azuryyyu@gmail.com] 
发送时间: 2013年4月16日 10:01
收件人: user@hadoop.apache.org
主题: Re: 答复: Region has been CLOSING for too long, this should eventually
complete or the server will expire, send RPC again

 

This is zookeeper issue.

 

please paste zookeeper log here. thanks.

 

On Tue, Apr 16, 2013 at 9:58 AM, dylan <dw...@gmail.com> wrote:

It is hbase-0.94.2-cdh4.2.0.

 

发件人: Ted Yu [mailto:yuzhihong@gmail.com] 
发送时间: 2013年4月16日 9:55
收件人: user@hbase.apache.org
主题: Re: Region has been CLOSING for too long, this should eventually
complete or the server will expire, send RPC again

 

I think this question would be more appropriate for HBase user mailing list.

 

Moving hadoop user to bcc.

 

Please tell us the HBase version you are using.

 

Thanks

On Mon, Apr 15, 2013 at 6:51 PM, dylan <dw...@gmail.com> wrote:

Hi

 

I am a newer for hadoop, and set up hadoop with tarball . I have 5 nodes for
cluster, 2 NN nodes with QJM (3 Journal Nodes, one of them on DN node.  ), 3
DN nodes with zookeepers,  It works fine.  When I reboot one data node
machine which includes zookeeper, after that , restart all processes. The
hadoop works fine, but hbase not. I cannot disable tables and drop tables.

 

The logs an follows:

The Hbase HMaster log:

DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Attempted to
unassign region -ROOT-,,0.70236052 but it is not currently assigned anywhere

,683 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in
transition timed out:  -ROOT-,,0.70236052 state=CLOSING, ts=1366001558865,
server=Master,60000,1366001238313

,683 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been
CLOSING for too long, this should eventually complete or the server will
expire, send RPC again

10,684 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting
unassignment of region -ROOT-,,0.70236052 (offlining)

 

The Hbase HRegionServer log:

 

DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=7.44 MB,
free=898.81 MB, max=906.24 MB, blocks=0, accesses=0, hits=0, hitRatio=0,
cachingAccesses=0, cachingHits=0, cachingHitsRatio=0, evictions=0,
evicted=0, evictedPerRun=NaN

 

The Hbase Web show：

Region                                              State

70236052    -ROOT-,,0.70236052 state=CLOSING, ts=Mon Apr 15 12:52:38 CST
2013 (75440s ago), server=Master,60000,1366001238313

 

How fix it?

 

Thanks.