You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Pankaj Misra <pa...@impetus.co.in> on 2012/10/17 17:35:00 UTC
HBase Issue after Namenode Formating
Hi All,
I have been using HBase 0.94.1 with Hadoop 0.23.1. I had the complete setup in a pseudo-distributed mode, and was running fine.
Today I had to reset and reformat the namenode and clear all the data as well to have a fresh run, but since then Hbase is not working as expected. For instance I am able to list the tables but cannot create new one, it reports that Master is initializing, as seen from the exception below
hbase(main):004:0> create 'testrun','samplecf'
ERROR: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:1626)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1154)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400)
In order to resolve this I have even tried to delete the root so that hbase can recreate it fresh. HBase is able to recreate the ROOT but this issue persists. I am also not able to connect via native java client, as it does not get reference of the Master.
Calling for help, as I think I am missing something here.
Thanks and Regards
Pankaj Misra
________________________________
Impetus Ranked in the Top 50 India's Best Companies to Work For 2012.
Impetus webcast 'Designing a Test Automation Framework for Multi-vendor Interoperable Systems' available at http://lf1.me/0E/.
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
Re: HBase Issue after Namenode Formating
Posted by lars hofhansl <lh...@yahoo.com>.
No problem...
We need to fix this in HBase too.There's HBASE-6294, but no patch for it, yet.
-- Lars
________________________________
From: Pankaj Misra <pa...@impetus.co.in>
To: "user@hbase.apache.org" <us...@hbase.apache.org>; lars hofhansl <lh...@yahoo.com>
Sent: Wednesday, October 17, 2012 3:18 PM
Subject: RE: HBase Issue after Namenode Formating
Hi Lars,
Thank you so much Lars for your perfect solution, it worked, you have been a life saver here, worked like a charm. Thanks again Lars.
Regards
Pankaj Misra
________________________________________
From: lars hofhansl [lhofhansl@yahoo.com]
Sent: Wednesday, October 17, 2012 11:43 PM
To: user@hbase.apache.org
Subject: Re: HBase Issue after Namenode Formating
Can you try clearing all the state in ZK as well? (stop ZK, wipe the ZK data directory, restart ZK)
----- Original Message -----
From: Pankaj Misra <pa...@impetus.co.in>
To: "user@hbase.apache.org" <us...@hbase.apache.org>
Cc:
Sent: Wednesday, October 17, 2012 11:09 AM
Subject: RE: HBase Issue after Namenode Formating
Hi All,
I did some more fact finding around this issue, and it seems to me that the Master is not getting initialized and hence I am not able to create table. I tried connecting via the Java client as well, which gave me the following exception. I am not sure what is causing Master to not initialize at all. I am not able to figure out, where am I going wrong, and would request help. Thanks.
java.lang.RuntimeException: java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1350496120102/Put/vlen=15/ts=0, .META.,,1/info:serverstartcode/1350496120102/Put/vlen=8/ts=0}
at com.benchmark.uds.hbase.NativeJavaHBaseLoader.createSchema(NativeJavaHBaseLoader.java:56)
at com.benchmark.uds.hbase.RequestGenerator.main(RequestGenerator.java:72)
Caused by: java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1350496120102/Put/vlen=15/ts=0, .META.,,1/info:serverstartcode/1350496120102/Put/vlen=8/ts=0}
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:985)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:172)
at org.apache.hadoop.hbase.catalog.MetaReader.getHTable(MetaReader.java:200)
at org.apache.hadoop.hbase.catalog.MetaReader.getMetaHTable(MetaReader.java:226)
at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:705)
at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:183)
at org.apache.hadoop.hbase.catalog.MetaReader.tableExists(MetaReader.java:448)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:233)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:247)
at com.benchmark.uds.hbase.NativeJavaHBaseLoader.createSchema(NativeJavaHBaseLoader.java:44)
Also, the error as seen in zookeeper log for the above session is
2012-10-17 23:32:04,274 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:51602 which had sessionid 0x13a6fd74b50000c
2012-10-17 23:32:05,105 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x13a6fd74b500009, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224)
at java.lang.Thread.run(Thread.java:662)
2012-10-17 23:32:05,105 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:51515 which had sessionid 0x13a6fd74b500009
Thanks and Regards
Pankaj Misra
________________________________________
From: Pankaj Misra
Sent: Wednesday, October 17, 2012 10:30 PM
To: user@hbase.apache.org
Subject: RE: HBase Issue after Namenode Formating
Hi All,
I have been using HBase 0.94.1 with Hadoop 0.23.1. I had the complete setup in a pseudo-distributed mode, and was running fine.
Today I had to reset and reformat the namenode and clear all the data as well to have a fresh run, but since then Hbase is not working as expected. For instance I am able to list the tables but cannot create new one, it reports that Master is initializing, as seen from the exception below
hbase(main):004:0> create 'testrun','samplecf'
ERROR: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:1626)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1154)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400)
In order to resolve this I have even tried to delete the root so that hbase can recreate it fresh. HBase is able to recreate the ROOT but this issue persists. I am also not able to connect via native java client, as it does not get reference of the Master.
Additionally I see the following error in the HBase Master logs
2012-10-17 21:15:33,043 ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master already exists and this is not a retry
2012-10-17 21:15:33,045 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Adding ZNode for /hbase/backup-masters/localhost,60000,1350488732582 in backup master directory
2012-10-17 21:15:33,083 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Current master has this master's address, localhost,60000,1350487401859; master was restarted? Deleting node.
2012-10-17 21:15:33,084 DEBUG org.apache.hadoop.hbase.master.ActiveMasterManager: No master available. Notifying waiting threads
2012-10-17 21:15:33,084 INFO org.apache.hadoop.hbase.master.HMaster: Cluster went down before this master became active
2012-10-17 21:15:33,084 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping service threads
Calling for help, as I think I am missing something here.
Thanks and Regards
Pankaj Misra
________________________________
Impetus Ranked in the Top 50 India's Best Companies to Work For 2012.
Impetus webcast 'Designing a Test Automation Framework for Multi-vendor Interoperable Systems' available at http://lf1.me/0E/.
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
Impetus Ranked in the Top 50 India’s Best Companies to Work For 2012.
Impetus webcast ‘Designing a Test Automation Framework for Multi-vendor Interoperable Systems’ available at http://lf1.me/0E/.
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
Impetus Ranked in the Top 50 India’s Best Companies to Work For 2012.
Impetus webcast ‘Designing a Test Automation Framework for Multi-vendor Interoperable Systems’ available at http://lf1.me/0E/.
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: HBase Issue after Namenode Formating
Posted by Pankaj Misra <pa...@impetus.co.in>.
Hi Lars,
Thank you so much Lars for your perfect solution, it worked, you have been a life saver here, worked like a charm. Thanks again Lars.
Regards
Pankaj Misra
________________________________________
From: lars hofhansl [lhofhansl@yahoo.com]
Sent: Wednesday, October 17, 2012 11:43 PM
To: user@hbase.apache.org
Subject: Re: HBase Issue after Namenode Formating
Can you try clearing all the state in ZK as well? (stop ZK, wipe the ZK data directory, restart ZK)
----- Original Message -----
From: Pankaj Misra <pa...@impetus.co.in>
To: "user@hbase.apache.org" <us...@hbase.apache.org>
Cc:
Sent: Wednesday, October 17, 2012 11:09 AM
Subject: RE: HBase Issue after Namenode Formating
Hi All,
I did some more fact finding around this issue, and it seems to me that the Master is not getting initialized and hence I am not able to create table. I tried connecting via the Java client as well, which gave me the following exception. I am not sure what is causing Master to not initialize at all. I am not able to figure out, where am I going wrong, and would request help. Thanks.
java.lang.RuntimeException: java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1350496120102/Put/vlen=15/ts=0, .META.,,1/info:serverstartcode/1350496120102/Put/vlen=8/ts=0}
at com.benchmark.uds.hbase.NativeJavaHBaseLoader.createSchema(NativeJavaHBaseLoader.java:56)
at com.benchmark.uds.hbase.RequestGenerator.main(RequestGenerator.java:72)
Caused by: java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1350496120102/Put/vlen=15/ts=0, .META.,,1/info:serverstartcode/1350496120102/Put/vlen=8/ts=0}
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:985)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:172)
at org.apache.hadoop.hbase.catalog.MetaReader.getHTable(MetaReader.java:200)
at org.apache.hadoop.hbase.catalog.MetaReader.getMetaHTable(MetaReader.java:226)
at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:705)
at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:183)
at org.apache.hadoop.hbase.catalog.MetaReader.tableExists(MetaReader.java:448)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:233)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:247)
at com.benchmark.uds.hbase.NativeJavaHBaseLoader.createSchema(NativeJavaHBaseLoader.java:44)
Also, the error as seen in zookeeper log for the above session is
2012-10-17 23:32:04,274 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:51602 which had sessionid 0x13a6fd74b50000c
2012-10-17 23:32:05,105 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x13a6fd74b500009, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224)
at java.lang.Thread.run(Thread.java:662)
2012-10-17 23:32:05,105 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:51515 which had sessionid 0x13a6fd74b500009
Thanks and Regards
Pankaj Misra
________________________________________
From: Pankaj Misra
Sent: Wednesday, October 17, 2012 10:30 PM
To: user@hbase.apache.org
Subject: RE: HBase Issue after Namenode Formating
Hi All,
I have been using HBase 0.94.1 with Hadoop 0.23.1. I had the complete setup in a pseudo-distributed mode, and was running fine.
Today I had to reset and reformat the namenode and clear all the data as well to have a fresh run, but since then Hbase is not working as expected. For instance I am able to list the tables but cannot create new one, it reports that Master is initializing, as seen from the exception below
hbase(main):004:0> create 'testrun','samplecf'
ERROR: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:1626)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1154)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400)
In order to resolve this I have even tried to delete the root so that hbase can recreate it fresh. HBase is able to recreate the ROOT but this issue persists. I am also not able to connect via native java client, as it does not get reference of the Master.
Additionally I see the following error in the HBase Master logs
2012-10-17 21:15:33,043 ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master already exists and this is not a retry
2012-10-17 21:15:33,045 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Adding ZNode for /hbase/backup-masters/localhost,60000,1350488732582 in backup master directory
2012-10-17 21:15:33,083 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Current master has this master's address, localhost,60000,1350487401859; master was restarted? Deleting node.
2012-10-17 21:15:33,084 DEBUG org.apache.hadoop.hbase.master.ActiveMasterManager: No master available. Notifying waiting threads
2012-10-17 21:15:33,084 INFO org.apache.hadoop.hbase.master.HMaster: Cluster went down before this master became active
2012-10-17 21:15:33,084 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping service threads
Calling for help, as I think I am missing something here.
Thanks and Regards
Pankaj Misra
________________________________
Impetus Ranked in the Top 50 India's Best Companies to Work For 2012.
Impetus webcast 'Designing a Test Automation Framework for Multi-vendor Interoperable Systems' available at http://lf1.me/0E/.
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
Impetus Ranked in the Top 50 India’s Best Companies to Work For 2012.
Impetus webcast ‘Designing a Test Automation Framework for Multi-vendor Interoperable Systems’ available at http://lf1.me/0E/.
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
Impetus Ranked in the Top 50 India’s Best Companies to Work For 2012.
Impetus webcast ‘Designing a Test Automation Framework for Multi-vendor Interoperable Systems’ available at http://lf1.me/0E/.
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
Re: HBase Issue after Namenode Formating
Posted by lars hofhansl <lh...@yahoo.com>.
Can you try clearing all the state in ZK as well? (stop ZK, wipe the ZK data directory, restart ZK)
----- Original Message -----
From: Pankaj Misra <pa...@impetus.co.in>
To: "user@hbase.apache.org" <us...@hbase.apache.org>
Cc:
Sent: Wednesday, October 17, 2012 11:09 AM
Subject: RE: HBase Issue after Namenode Formating
Hi All,
I did some more fact finding around this issue, and it seems to me that the Master is not getting initialized and hence I am not able to create table. I tried connecting via the Java client as well, which gave me the following exception. I am not sure what is causing Master to not initialize at all. I am not able to figure out, where am I going wrong, and would request help. Thanks.
java.lang.RuntimeException: java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1350496120102/Put/vlen=15/ts=0, .META.,,1/info:serverstartcode/1350496120102/Put/vlen=8/ts=0}
at com.benchmark.uds.hbase.NativeJavaHBaseLoader.createSchema(NativeJavaHBaseLoader.java:56)
at com.benchmark.uds.hbase.RequestGenerator.main(RequestGenerator.java:72)
Caused by: java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1350496120102/Put/vlen=15/ts=0, .META.,,1/info:serverstartcode/1350496120102/Put/vlen=8/ts=0}
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:985)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:172)
at org.apache.hadoop.hbase.catalog.MetaReader.getHTable(MetaReader.java:200)
at org.apache.hadoop.hbase.catalog.MetaReader.getMetaHTable(MetaReader.java:226)
at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:705)
at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:183)
at org.apache.hadoop.hbase.catalog.MetaReader.tableExists(MetaReader.java:448)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:233)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:247)
at com.benchmark.uds.hbase.NativeJavaHBaseLoader.createSchema(NativeJavaHBaseLoader.java:44)
Also, the error as seen in zookeeper log for the above session is
2012-10-17 23:32:04,274 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:51602 which had sessionid 0x13a6fd74b50000c
2012-10-17 23:32:05,105 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x13a6fd74b500009, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224)
at java.lang.Thread.run(Thread.java:662)
2012-10-17 23:32:05,105 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:51515 which had sessionid 0x13a6fd74b500009
Thanks and Regards
Pankaj Misra
________________________________________
From: Pankaj Misra
Sent: Wednesday, October 17, 2012 10:30 PM
To: user@hbase.apache.org
Subject: RE: HBase Issue after Namenode Formating
Hi All,
I have been using HBase 0.94.1 with Hadoop 0.23.1. I had the complete setup in a pseudo-distributed mode, and was running fine.
Today I had to reset and reformat the namenode and clear all the data as well to have a fresh run, but since then Hbase is not working as expected. For instance I am able to list the tables but cannot create new one, it reports that Master is initializing, as seen from the exception below
hbase(main):004:0> create 'testrun','samplecf'
ERROR: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:1626)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1154)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400)
In order to resolve this I have even tried to delete the root so that hbase can recreate it fresh. HBase is able to recreate the ROOT but this issue persists. I am also not able to connect via native java client, as it does not get reference of the Master.
Additionally I see the following error in the HBase Master logs
2012-10-17 21:15:33,043 ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master already exists and this is not a retry
2012-10-17 21:15:33,045 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Adding ZNode for /hbase/backup-masters/localhost,60000,1350488732582 in backup master directory
2012-10-17 21:15:33,083 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Current master has this master's address, localhost,60000,1350487401859; master was restarted? Deleting node.
2012-10-17 21:15:33,084 DEBUG org.apache.hadoop.hbase.master.ActiveMasterManager: No master available. Notifying waiting threads
2012-10-17 21:15:33,084 INFO org.apache.hadoop.hbase.master.HMaster: Cluster went down before this master became active
2012-10-17 21:15:33,084 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping service threads
Calling for help, as I think I am missing something here.
Thanks and Regards
Pankaj Misra
________________________________
Impetus Ranked in the Top 50 India's Best Companies to Work For 2012.
Impetus webcast 'Designing a Test Automation Framework for Multi-vendor Interoperable Systems' available at http://lf1.me/0E/.
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
Impetus Ranked in the Top 50 India’s Best Companies to Work For 2012.
Impetus webcast ‘Designing a Test Automation Framework for Multi-vendor Interoperable Systems’ available at http://lf1.me/0E/.
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: HBase Issue after Namenode Formating
Posted by Pankaj Misra <pa...@impetus.co.in>.
Hi All,
I did some more fact finding around this issue, and it seems to me that the Master is not getting initialized and hence I am not able to create table. I tried connecting via the Java client as well, which gave me the following exception. I am not sure what is causing Master to not initialize at all. I am not able to figure out, where am I going wrong, and would request help. Thanks.
java.lang.RuntimeException: java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1350496120102/Put/vlen=15/ts=0, .META.,,1/info:serverstartcode/1350496120102/Put/vlen=8/ts=0}
at com.benchmark.uds.hbase.NativeJavaHBaseLoader.createSchema(NativeJavaHBaseLoader.java:56)
at com.benchmark.uds.hbase.RequestGenerator.main(RequestGenerator.java:72)
Caused by: java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1350496120102/Put/vlen=15/ts=0, .META.,,1/info:serverstartcode/1350496120102/Put/vlen=8/ts=0}
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:985)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:172)
at org.apache.hadoop.hbase.catalog.MetaReader.getHTable(MetaReader.java:200)
at org.apache.hadoop.hbase.catalog.MetaReader.getMetaHTable(MetaReader.java:226)
at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:705)
at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:183)
at org.apache.hadoop.hbase.catalog.MetaReader.tableExists(MetaReader.java:448)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:233)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:247)
at com.benchmark.uds.hbase.NativeJavaHBaseLoader.createSchema(NativeJavaHBaseLoader.java:44)
Also, the error as seen in zookeeper log for the above session is
2012-10-17 23:32:04,274 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:51602 which had sessionid 0x13a6fd74b50000c
2012-10-17 23:32:05,105 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x13a6fd74b500009, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224)
at java.lang.Thread.run(Thread.java:662)
2012-10-17 23:32:05,105 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:51515 which had sessionid 0x13a6fd74b500009
Thanks and Regards
Pankaj Misra
________________________________________
From: Pankaj Misra
Sent: Wednesday, October 17, 2012 10:30 PM
To: user@hbase.apache.org
Subject: RE: HBase Issue after Namenode Formating
Hi All,
I have been using HBase 0.94.1 with Hadoop 0.23.1. I had the complete setup in a pseudo-distributed mode, and was running fine.
Today I had to reset and reformat the namenode and clear all the data as well to have a fresh run, but since then Hbase is not working as expected. For instance I am able to list the tables but cannot create new one, it reports that Master is initializing, as seen from the exception below
hbase(main):004:0> create 'testrun','samplecf'
ERROR: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:1626)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1154)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400)
In order to resolve this I have even tried to delete the root so that hbase can recreate it fresh. HBase is able to recreate the ROOT but this issue persists. I am also not able to connect via native java client, as it does not get reference of the Master.
Additionally I see the following error in the HBase Master logs
2012-10-17 21:15:33,043 ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master already exists and this is not a retry
2012-10-17 21:15:33,045 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Adding ZNode for /hbase/backup-masters/localhost,60000,1350488732582 in backup master directory
2012-10-17 21:15:33,083 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Current master has this master's address, localhost,60000,1350487401859; master was restarted? Deleting node.
2012-10-17 21:15:33,084 DEBUG org.apache.hadoop.hbase.master.ActiveMasterManager: No master available. Notifying waiting threads
2012-10-17 21:15:33,084 INFO org.apache.hadoop.hbase.master.HMaster: Cluster went down before this master became active
2012-10-17 21:15:33,084 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping service threads
Calling for help, as I think I am missing something here.
Thanks and Regards
Pankaj Misra
________________________________
Impetus Ranked in the Top 50 India's Best Companies to Work For 2012.
Impetus webcast 'Designing a Test Automation Framework for Multi-vendor Interoperable Systems' available at http://lf1.me/0E/.
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
Impetus Ranked in the Top 50 India’s Best Companies to Work For 2012.
Impetus webcast ‘Designing a Test Automation Framework for Multi-vendor Interoperable Systems’ available at http://lf1.me/0E/.
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: HBase Issue after Namenode Formating
Posted by Pankaj Misra <pa...@impetus.co.in>.
Hi All,
I have been using HBase 0.94.1 with Hadoop 0.23.1. I had the complete setup in a pseudo-distributed mode, and was running fine.
Today I had to reset and reformat the namenode and clear all the data as well to have a fresh run, but since then Hbase is not working as expected. For instance I am able to list the tables but cannot create new one, it reports that Master is initializing, as seen from the exception below
hbase(main):004:0> create 'testrun','samplecf'
ERROR: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:1626)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1154)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400)
In order to resolve this I have even tried to delete the root so that hbase can recreate it fresh. HBase is able to recreate the ROOT but this issue persists. I am also not able to connect via native java client, as it does not get reference of the Master.
Additionally I see the following error in the HBase Master logs
2012-10-17 21:15:33,043 ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master already exists and this is not a retry
2012-10-17 21:15:33,045 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Adding ZNode for /hbase/backup-masters/localhost,60000,1350488732582 in backup master directory
2012-10-17 21:15:33,083 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Current master has this master's address, localhost,60000,1350487401859; master was restarted? Deleting node.
2012-10-17 21:15:33,084 DEBUG org.apache.hadoop.hbase.master.ActiveMasterManager: No master available. Notifying waiting threads
2012-10-17 21:15:33,084 INFO org.apache.hadoop.hbase.master.HMaster: Cluster went down before this master became active
2012-10-17 21:15:33,084 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping service threads
Calling for help, as I think I am missing something here.
Thanks and Regards
Pankaj Misra
________________________________
Impetus Ranked in the Top 50 India's Best Companies to Work For 2012.
Impetus webcast 'Designing a Test Automation Framework for Multi-vendor Interoperable Systems' available at http://lf1.me/0E/.
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: HBase Issue after Namenode Formating
Posted by Pankaj Misra <pa...@impetus.co.in>.
Additionally I see the following error in the HBase Master logs
2012-10-17 21:15:33,043 ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master already exists and this is not a retry
2012-10-17 21:15:33,045 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Adding ZNode for /hbase/backup-masters/localhost,60000,1350488732582 in backup master directory
2012-10-17 21:15:33,083 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Current master has this master's address, localhost,60000,1350487401859; master was restarted? Deleting node.
2012-10-17 21:15:33,084 DEBUG org.apache.hadoop.hbase.master.ActiveMasterManager: No master available. Notifying waiting threads
2012-10-17 21:15:33,084 INFO org.apache.hadoop.hbase.master.HMaster: Cluster went down before this master became active
2012-10-17 21:15:33,084 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping service threads
Thanks and Regards
Pankaj Misra
________________________________
From: Pankaj Misra
Sent: Wednesday, October 17, 2012 9:05 PM
To: user@hbase.apache.org
Subject: HBase Issue after Namenode Formating
Hi All,
I have been using HBase 0.94.1 with Hadoop 0.23.1. I had the complete setup in a pseudo-distributed mode, and was running fine.
Today I had to reset and reformat the namenode and clear all the data as well to have a fresh run, but since then Hbase is not working as expected. For instance I am able to list the tables but cannot create new one, it reports that Master is initializing, as seen from the exception below
hbase(main):004:0> create 'testrun','samplecf'
ERROR: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:1626)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1154)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400)
In order to resolve this I have even tried to delete the root so that hbase can recreate it fresh. HBase is able to recreate the ROOT but this issue persists. I am also not able to connect via native java client, as it does not get reference of the Master.
Calling for help, as I think I am missing something here.
Thanks and Regards
Pankaj Misra
________________________________
Impetus Ranked in the Top 50 India's Best Companies to Work For 2012.
Impetus webcast 'Designing a Test Automation Framework for Multi-vendor Interoperable Systems' available at http://lf1.me/0E/.
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.