You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by techbuddy <te...@gmail.com> on 2012/10/11 23:11:48 UTC
[Stand alone - distributed mode] HBase master isn't initializing
completely
Hi,
I've a standalone Hbase cluster configured in a distributed mode (i.e. the
ZK, Master and the RegServer all running in separate JVMs on the same host).
The HBase master doesn't seem to be intializing successfully. This has
started happening since I encountered the stop-hbase script going into an
indefinite loop followed by a forced kill of the three Hbase processes
(Zk,Master and Region server). I did manually kill these processes couple of
times earlier too, but the Hbase master never stalled partially initialized.
I have also removed the zookeeper and the hbase/test/hbase folders in an
effort to clean things up before triggering the start-hbase script.
Symptoms
1. The create table command on the Hbase shell errors out with a
PleaseHoldException (Master in initializing)
2. On the Admin UI, the regon server lists only the ROOT and the single META
regions. It doesn't list one single region of the user tables. However ,a
list command on the shell indeed lists all the user tables (that were
created in the past)
3. On the Master logs, I see the following locate region failures and
retries on the META table
2012-10-11 14:09:03,970 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-,
metaLocation={region=-ROOT-,,0.70236052,
hostname=materialstranger.corp.XXX.com, port=60020}, attempt=77 of 100
failed; retrying after sleep of 32172 because: Connection refused
2012-10-11 14:09:03,971 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1bbbafc;
serverName=materialstranger.corp.XXX.com,60020,1349987514876
2012-10-11 14:09:36,197 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1bbbafc;
serverName=materialstranger.corp.XXX.com,60020,1349987514876
2012-10-11 14:09:36,197 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-,
metaLocation={region=-ROOT-,,0.70236052,
hostname=materialstranger.corp.XXX.com, port=60020}, attempt=78 of 100
failed; retrying after sleep of 32275 because: Connection refused
2012-10-11 14:09:36,198 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1bbbafc;
serverName=materialstranger.corp.XXX.com,60020,1349987514876
Any pointers on how to restore the cluster to a clean ,working state would
be highly appreciated.
-SB
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Stand-alone-distributed-mode-HBase-master-isn-t-initializing-completely-tp4032869.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: 答复: [Stand alone - distributed mode] HBase master isn't initializing completely
Posted by Suraj Varma <sv...@gmail.com>.
Shutdown the cluster and remove all the *.pid files from the
configured tmp.dir - especially of the master. Then bring up the
cluster again. See if this resolves it.
Also - did you check your hosts file and refer to the host
configuration specified in the hbase online guide?
--S
On Fri, Oct 12, 2012 at 12:27 AM, techbuddy <te...@gmail.com> wrote:
> I'm using the local filesystem for the underlying filesystem of hbase, and
> the Hbase version is 0.94.0
>
>
>
> --
> View this message in context: http://apache-hbase.679495.n3.nabble.com/Stand-alone-distributed-mode-HBase-master-isn-t-initializing-completely-tp4032869p4032891.html
> Sent from the HBase User mailing list archive at Nabble.com.
Re: 答复: [Stand alone - distributed mode] HBase master isn't initializing completely
Posted by techbuddy <te...@gmail.com>.
I'm using the local filesystem for the underlying filesystem of hbase, and
the Hbase version is 0.94.0
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Stand-alone-distributed-mode-HBase-master-isn-t-initializing-completely-tp4032869p4032891.html
Sent from the HBase User mailing list archive at Nabble.com.
答复: [Stand alone - distributed mode] HBase master isn't initializing completely
Posted by 谢良 <xi...@xiaomi.com>.
Is there any WARNING/ERROR in HDFS logfile ?
Please ensure zk&hdfs are in healthy status firstly
Could you provide your hbse version, it'll be great:)
________________________________________
发件人: techbuddy [techbuddyblr@gmail.com]
发送时间: 2012年10月12日 5:11
收件人: user@hbase.apache.org
主题: [Stand alone - distributed mode] HBase master isn't initializing completely
Hi,
I've a standalone Hbase cluster configured in a distributed mode (i.e. the
ZK, Master and the RegServer all running in separate JVMs on the same host).
The HBase master doesn't seem to be intializing successfully. This has
started happening since I encountered the stop-hbase script going into an
indefinite loop followed by a forced kill of the three Hbase processes
(Zk,Master and Region server). I did manually kill these processes couple of
times earlier too, but the Hbase master never stalled partially initialized.
I have also removed the zookeeper and the hbase/test/hbase folders in an
effort to clean things up before triggering the start-hbase script.
Symptoms
1. The create table command on the Hbase shell errors out with a
PleaseHoldException (Master in initializing)
2. On the Admin UI, the regon server lists only the ROOT and the single META
regions. It doesn't list one single region of the user tables. However ,a
list command on the shell indeed lists all the user tables (that were
created in the past)
3. On the Master logs, I see the following locate region failures and
retries on the META table
2012-10-11 14:09:03,970 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-,
metaLocation={region=-ROOT-,,0.70236052,
hostname=materialstranger.corp.XXX.com, port=60020}, attempt=77 of 100
failed; retrying after sleep of 32172 because: Connection refused
2012-10-11 14:09:03,971 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1bbbafc;
serverName=materialstranger.corp.XXX.com,60020,1349987514876
2012-10-11 14:09:36,197 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1bbbafc;
serverName=materialstranger.corp.XXX.com,60020,1349987514876
2012-10-11 14:09:36,197 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-,
metaLocation={region=-ROOT-,,0.70236052,
hostname=materialstranger.corp.XXX.com, port=60020}, attempt=78 of 100
failed; retrying after sleep of 32275 because: Connection refused
2012-10-11 14:09:36,198 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1bbbafc;
serverName=materialstranger.corp.XXX.com,60020,1349987514876
Any pointers on how to restore the cluster to a clean ,working state would
be highly appreciated.
-SB
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Stand-alone-distributed-mode-HBase-master-isn-t-initializing-completely-tp4032869.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: [Stand alone - distributed mode] HBase master isn't
initializing completely
Posted by techbuddy <te...@gmail.com>.
We root caused the problem to corruption of the .META. table data,that was
being caused by a bug in one of our co-processors.
The co-processor was returning a null internalScanner in one of the
RegionObserver callbacks.
I'd suggest you run the hbck tool, that does an hbase file system check, if
your cluster is starting up. If not, try running the OfflineMetaRepair
tool. Also, visit the master and region server logs for more information.
On Sat, Jun 29, 2013 at 9:59 PM, abhishek1015 [via Apache HBase] <
ml-node+s679495n4047146h3@n3.nabble.com> wrote:
> I am facing same problem.
>
> @techbuddy: Is this problem resolved?
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-hbase.679495.n3.nabble.com/Stand-alone-distributed-mode-HBase-master-isn-t-initializing-completely-tp4032869p4047146.html
> To unsubscribe from [Stand alone - distributed mode] HBase master isn't
> initializing completely, click here<http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4032869&code=dGVjaGJ1ZGR5YmxyQGdtYWlsLmNvbXw0MDMyODY5fC01ODkyOTcwMzc=>
> .
> NAML<http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Stand-alone-distributed-mode-HBase-master-isn-t-initializing-completely-tp4032869p4047147.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: [Stand alone - distributed mode] HBase master isn't
initializing completely
Posted by abhishek1015 <ab...@gmail.com>.
I am facing same problem.
@techbuddy: Is this problem resolved?
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Stand-alone-distributed-mode-HBase-master-isn-t-initializing-completely-tp4032869p4047146.html
Sent from the HBase User mailing list archive at Nabble.com.