You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Irwan Hadi <ih...@gmail.com> on 2012/07/25 08:28:45 UTC
Issue with hbase
A few weeks ago we installed hbase 0.90.5 on our dev system, to play
around with it, along with Hadoop, and HDFS. All of these are
installed on a Centos 6 server.
Even though we specified hbase to use HDFS in hbase-site.xml, we had a
small configuration typo which then apparently caused it to default
storing data to /tmp/hbase-hadoop/hbase
As expected, after running for a couple days, the tmpwatch cron
cleaned up certain files in /tmp. The issue wasn't apparent until
hbase had to be cycled, which shows that we have 0 tables.
The question now is:
We still have the tables intact, but lost the META, ROOT, and
.regioninfo under /tmp/hbase-hadoop/hbase/<table_name>/id/<number> ,
are these enough to somehow recover the tables, if so how can we do
this?
We tried hbase hbck already, but it does not want to fix the issue
=======================================================================================
12/07/25 06:23:11 INFO zookeeper.ClientCnxn: EventThread shut down
Number of Tables: 0
Number of live region servers: 1
Number of dead region servers: 0
Number of empty REGIONINFO_QUALIFIER rows in .META.: 0
12/07/25 06:23:11 INFO zookeeper.ZooKeeper: Initiating client
connection, connectString=localhost:2181 sessionTimeout=180000
watcher=hconnection
12/07/25 06:23:11 INFO zookeeper.ClientCnxn: Opening socket connection
to server localhost/127.0.0.1:2181
12/07/25 06:23:11 INFO zookeeper.ClientCnxn: Socket connection
established to localhost/127.0.0.1:2181, initiating session
12/07/25 06:23:11 INFO zookeeper.ClientCnxn: Session establishment
complete on server localhost/127.0.0.1:2181, sessionid =
0x138bca7fff5000a, negotiated timeout = 40000
12/07/25 06:23:11 INFO
client.HConnectionManager$HConnectionImplementation: Closed zookeeper
sessionid=0x138bca7fff5000a
12/07/25 06:23:11 INFO zookeeper.ZooKeeper: Session: 0x138bca7fff5000a closed
12/07/25 06:23:11 INFO zookeeper.ClientCnxn: EventThread shut down
12/07/25 06:23:11 DEBUG
client.HConnectionManager$HConnectionImplementation: The connection to
null has been closed.
ERROR: Region file:/tmp/hbase-hadoop/hbase/news/349a8254e3fe134eb0ec82693e8ff905
on HDFS, but not listed in META or deployed on any region server
ERROR: Region file:/tmp/hbase-hadoop/hbase/story/89770459431a8e7558d952fc4ee7c60b
on HDFS, but not listed in META or deployed on any region server
ERROR: Region file:/tmp/hbase-hadoop/hbase/topic/cb411aee8e80c70b03b0ebdc574ba8ad
on HDFS, but not listed in META or deployed on any region server
Summary:
-ROOT- is okay.
Number of regions: 1
Deployed on: web01.dev.:52768
.META. is okay.
Number of regions: 1
Deployed on: web01.dev.:52768
3 inconsistencies detected.
Status: INCONSISTENT
=======================================================================================
We also tried rebuilding the META using the following "hbase
org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair -base
file:///tmp/hbase-hadoop/hbase/" , but got the following error:
=======================================================================================
12/07/25 05:25:58 INFO util.HBaseFsck: Loading HBase regioninfo from HDFS...
12/07/25 05:25:59 DEBUG util.HBaseFsck: HRegionInfo read: REGION =>
{NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED =>
1028785192, TABLE => {{NAME => '.META.', IS_META => 'true', FAMILIES
=> [{NAME => 'info', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0',
VERSIONS => '10', COMPRESSION => 'NONE', TTL => '2147483647',
BLOCKSIZE => '8192', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}
ERROR: Unable to load region info for table news! It may be an
invalid format or version file. You may want to remove
file:/tmp/hbase-hadoop/hbase/news/349a8254e3fe134eb0ec82693e8ff905
region from hdfs and retry.
12/07/25 05:25:59 ERROR util.HBaseFsck: Unable to load region info for
table news! It may be an invalid format or version file. You may
want to remove file:/tmp/hbase-hadoop/hbase/news/349a8254e3fe134eb0ec82693e8ff905
region from hdfs and retry.
java.io.FileNotFoundException: File
file:/tmp/hbase-hadoop/hbase/news/349a8254e3fe134eb0ec82693e8ff905/.regioninfo
does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:256)
at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:284)
at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:402)
at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:90)
12/07/25 05:25:59 DEBUG util.HBaseFsck: HRegionInfo read: REGION =>
{NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED =>
70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META =>
'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'NONE',
REPLICATION_SCOPE => '0', VERSIONS => '10', COMPRESSION => 'NONE', TTL
=> '2147483647', BLOCKSIZE => '8192', IN_MEMORY => 'false', BLOCKCACHE
=> 'false'}]}}
ERROR: Unable to load region info for table story! It may be an
invalid format or version file. You may want to remove
file:/tmp/hbase-hadoop/hbase/story/89770459431a8e7558d952fc4ee7c60b
region from hdfs and retry.
12/07/25 05:25:59 ERROR util.HBaseFsck: Unable to load region info for
table story! It may be an invalid format or version file. You may
want to remove file:/tmp/hbase-hadoop/hbase/story/89770459431a8e7558d952fc4ee7c60b
region from hdfs and retry.
java.io.FileNotFoundException: File
file:/tmp/hbase-hadoop/hbase/story/89770459431a8e7558d952fc4ee7c60b/.regioninfo
does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:256)
at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:284)
at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:402)
at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:90)
ERROR: Unable to load region info for table topic! It may be an
invalid format or version file. You may want to remove
file:/tmp/hbase-hadoop/hbase/topic/cb411aee8e80c70b03b0ebdc574ba8ad
region from hdfs and retry.
12/07/25 05:25:59 ERROR util.HBaseFsck: Unable to load region info for
table topic! It may be an invalid format or version file. You may
want to remove file:/tmp/hbase-hadoop/hbase/topic/cb411aee8e80c70b03b0ebdc574ba8ad
region from hdfs and retry.
java.io.FileNotFoundException: File
file:/tmp/hbase-hadoop/hbase/topic/cb411aee8e80c70b03b0ebdc574ba8ad/.regioninfo
does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:256)
at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:284)
at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:402)
at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:90)
12/07/25 05:25:59 ERROR util.HBaseFsck: Bailed out due to:
org.apache.hadoop.hbase.util.HBaseFsck$RegionInfoLoadException: Unable
to load region info for table news! It may be an invalid format or
version file. You may want to remove
file:/tmp/hbase-hadoop/hbase/news/349a8254e3fe134eb0ec82693e8ff905
region from hdfs and retry.
at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:292)
at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:402)
at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:90)
12/07/25 05:25:59 ERROR util.HBaseFsck: Bailed out due to:
=======================================================================================
Re: Issue with hbase
Posted by Kevin O'dell <ke...@cloudera.com>.
Irwan,
Just to be clear: You don't have the META, ROOT, or any .regioninfo
files, but the storefiles are still intact?
If so, you should be able to recreate your tables using the correct HBase
home and bulk load your data back in? I think that would work, but I will
defer to Stack or Michael on this one.
On Wed, Jul 25, 2012 at 2:28 AM, Irwan Hadi <ih...@gmail.com> wrote:
> A few weeks ago we installed hbase 0.90.5 on our dev system, to play
> around with it, along with Hadoop, and HDFS. All of these are
> installed on a Centos 6 server.
> Even though we specified hbase to use HDFS in hbase-site.xml, we had a
> small configuration typo which then apparently caused it to default
> storing data to /tmp/hbase-hadoop/hbase
>
> As expected, after running for a couple days, the tmpwatch cron
> cleaned up certain files in /tmp. The issue wasn't apparent until
> hbase had to be cycled, which shows that we have 0 tables.
>
> The question now is:
> We still have the tables intact, but lost the META, ROOT, and
> .regioninfo under /tmp/hbase-hadoop/hbase/<table_name>/id/<number> ,
> are these enough to somehow recover the tables, if so how can we do
> this?
> We tried hbase hbck already, but it does not want to fix the issue
>
> =======================================================================================
> 12/07/25 06:23:11 INFO zookeeper.ClientCnxn: EventThread shut down
> Number of Tables: 0
> Number of live region servers: 1
> Number of dead region servers: 0
> Number of empty REGIONINFO_QUALIFIER rows in .META.: 0
> 12/07/25 06:23:11 INFO zookeeper.ZooKeeper: Initiating client
> connection, connectString=localhost:2181 sessionTimeout=180000
> watcher=hconnection
> 12/07/25 06:23:11 INFO zookeeper.ClientCnxn: Opening socket connection
> to server localhost/127.0.0.1:2181
> 12/07/25 06:23:11 INFO zookeeper.ClientCnxn: Socket connection
> established to localhost/127.0.0.1:2181, initiating session
> 12/07/25 06:23:11 INFO zookeeper.ClientCnxn: Session establishment
> complete on server localhost/127.0.0.1:2181, sessionid =
> 0x138bca7fff5000a, negotiated timeout = 40000
> 12/07/25 06:23:11 INFO
> client.HConnectionManager$HConnectionImplementation: Closed zookeeper
> sessionid=0x138bca7fff5000a
> 12/07/25 06:23:11 INFO zookeeper.ZooKeeper: Session: 0x138bca7fff5000a
> closed
> 12/07/25 06:23:11 INFO zookeeper.ClientCnxn: EventThread shut down
> 12/07/25 06:23:11 DEBUG
> client.HConnectionManager$HConnectionImplementation: The connection to
> null has been closed.
> ERROR: Region
> file:/tmp/hbase-hadoop/hbase/news/349a8254e3fe134eb0ec82693e8ff905
> on HDFS, but not listed in META or deployed on any region server
> ERROR: Region
> file:/tmp/hbase-hadoop/hbase/story/89770459431a8e7558d952fc4ee7c60b
> on HDFS, but not listed in META or deployed on any region server
> ERROR: Region
> file:/tmp/hbase-hadoop/hbase/topic/cb411aee8e80c70b03b0ebdc574ba8ad
> on HDFS, but not listed in META or deployed on any region server
> Summary:
> -ROOT- is okay.
> Number of regions: 1
> Deployed on: web01.dev.:52768
> .META. is okay.
> Number of regions: 1
> Deployed on: web01.dev.:52768
> 3 inconsistencies detected.
> Status: INCONSISTENT
>
> =======================================================================================
>
>
>
> We also tried rebuilding the META using the following "hbase
> org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair -base
> file:///tmp/hbase-hadoop/hbase/" , but got the following error:
>
> =======================================================================================
> 12/07/25 05:25:58 INFO util.HBaseFsck: Loading HBase regioninfo from
> HDFS...
> 12/07/25 05:25:59 DEBUG util.HBaseFsck: HRegionInfo read: REGION =>
> {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED =>
> 1028785192, TABLE => {{NAME => '.META.', IS_META => 'true', FAMILIES
> => [{NAME => 'info', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0',
> VERSIONS => '10', COMPRESSION => 'NONE', TTL => '2147483647',
> BLOCKSIZE => '8192', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}
> ERROR: Unable to load region info for table news! It may be an
> invalid format or version file. You may want to remove
> file:/tmp/hbase-hadoop/hbase/news/349a8254e3fe134eb0ec82693e8ff905
> region from hdfs and retry.
> 12/07/25 05:25:59 ERROR util.HBaseFsck: Unable to load region info for
> table news! It may be an invalid format or version file. You may
> want to remove
> file:/tmp/hbase-hadoop/hbase/news/349a8254e3fe134eb0ec82693e8ff905
> region from hdfs and retry.
> java.io.FileNotFoundException: File
>
> file:/tmp/hbase-hadoop/hbase/news/349a8254e3fe134eb0ec82693e8ff905/.regioninfo
> does not exist.
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:256)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:284)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:402)
> at
> org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:90)
> 12/07/25 05:25:59 DEBUG util.HBaseFsck: HRegionInfo read: REGION =>
> {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED =>
> 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META =>
> 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'NONE',
> REPLICATION_SCOPE => '0', VERSIONS => '10', COMPRESSION => 'NONE', TTL
> => '2147483647', BLOCKSIZE => '8192', IN_MEMORY => 'false', BLOCKCACHE
> => 'false'}]}}
> ERROR: Unable to load region info for table story! It may be an
> invalid format or version file. You may want to remove
> file:/tmp/hbase-hadoop/hbase/story/89770459431a8e7558d952fc4ee7c60b
> region from hdfs and retry.
> 12/07/25 05:25:59 ERROR util.HBaseFsck: Unable to load region info for
> table story! It may be an invalid format or version file. You may
> want to remove
> file:/tmp/hbase-hadoop/hbase/story/89770459431a8e7558d952fc4ee7c60b
> region from hdfs and retry.
> java.io.FileNotFoundException: File
>
> file:/tmp/hbase-hadoop/hbase/story/89770459431a8e7558d952fc4ee7c60b/.regioninfo
> does not exist.
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:256)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:284)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:402)
> at
> org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:90)
> ERROR: Unable to load region info for table topic! It may be an
> invalid format or version file. You may want to remove
> file:/tmp/hbase-hadoop/hbase/topic/cb411aee8e80c70b03b0ebdc574ba8ad
> region from hdfs and retry.
> 12/07/25 05:25:59 ERROR util.HBaseFsck: Unable to load region info for
> table topic! It may be an invalid format or version file. You may
> want to remove
> file:/tmp/hbase-hadoop/hbase/topic/cb411aee8e80c70b03b0ebdc574ba8ad
> region from hdfs and retry.
> java.io.FileNotFoundException: File
>
> file:/tmp/hbase-hadoop/hbase/topic/cb411aee8e80c70b03b0ebdc574ba8ad/.regioninfo
> does not exist.
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:256)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:284)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:402)
> at
> org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:90)
> 12/07/25 05:25:59 ERROR util.HBaseFsck: Bailed out due to:
> org.apache.hadoop.hbase.util.HBaseFsck$RegionInfoLoadException: Unable
> to load region info for table news! It may be an invalid format or
> version file. You may want to remove
> file:/tmp/hbase-hadoop/hbase/news/349a8254e3fe134eb0ec82693e8ff905
> region from hdfs and retry.
> at
> org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:292)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:402)
> at
> org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:90)
> 12/07/25 05:25:59 ERROR util.HBaseFsck: Bailed out due to:
>
> =======================================================================================
>
--
Kevin O'Dell
Customer Operations Engineer, Cloudera