You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Arinto Murdopo <ar...@gmail.com> on 2015/02/23 08:47:34 UTC
HBase Region always in transition + corrupt HDFS
Hi all,
We're running HBase (0.94.15-cdh4.6.0) on top of HDFS (Hadoop
2.0.0-cdh4.6.0).
For all of our tables, we set the replication factor to 1 (dfs.replication
= 1 in hbase-site.xml). We set to 1 because we want to minimize the HDFS
usage (now we realize we should set this value to at least 2, because
"failure is a norm" in distributed systems).
Due to the amount of data, at some point, we have low disk space in HDFS
and one of our DNs was down. Now we have these problems in HBase and HDFS
although we have recovered our DN.
*Issue#1*. Some of HBase region always in transition. '*hbase hbck -repair*'
is stuck because it's waiting for region transition to finish. Some output
*hbase(main):003:0> status 'detailed'*
*12 regionsInTransition*
*
plr_id_insta_media_live,\x02:;6;7;398962:3:399a49:653:64,1421565172917.1528f288473632aca2636443574a6ba1.
state=OPENING, ts=1424227696897, server=null*
*
plr_sg_insta_media_live,\x0098;522:997;8798665a64;67879,1410768824800.2c79bbc5c0dc2d2b39c04c8abc0a90ff.
state=OFFLINE, ts=1424227714203, server=null*
*
plr_sg_insta_media_live,\x00465892:9935773828;a4459;649,1410767723471.55097cfc60bc9f50303dadb02abcd64b.
state=OPENING, ts=1424227701234, server=null*
*
plr_sg_insta_media_live,\x00474973488232837733a38744,1410767723471.740d6655afb74a2ff421c6ef16037f57.
state=OPENING, ts=1424227708053, server=null*
*
plr_id_insta_media_live,\x02::449::4;:466;3988a6432677;3,1419435100617.7caf3d749dce37037eec9ccc29d272a1.
state=OPENING, ts=1424227701484, server=null*
*
plr_sg_insta_media_live,\x05779793546323;::4:4a3:8227928,1418845792479.81c4da129ae5b7b204d5373d9e0fea3d.
state=OPENING, ts=1424227705353, server=null*
*
plr_sg_insta_media_live,\x009;5:686348963:33:5a5634887,1410769837567.8a9ded24960a7787ca016e2073b24151.
state=OPENING, ts=1424227706293, server=null*
*
plr_sg_insta_media_live,\x0375;6;7377578;84226a7663792,1418980694076.a1e1c98f646ee899010f19a9c693c67c.
state=OPENING, ts=1424227680569, server=null*
*
plr_sg_insta_media_live,\x018;3826368274679364a3;;73457;,1421425643816.b04ffda1b2024bac09c9e6246fb7b183.
state=OPENING, ts=1424227680538, server=null*
*
plr_sg_insta_media_live,\x0154752;22:43377542:a:86:239,1410771044924.c57d6b4d23f21d3e914a91721a99ce12.
state=OPENING, ts=1424227710847, server=null*
*
plr_sg_insta_media_live,\x0069;7;9384697:;8685a885485:,1410767928822.c7b5e53cdd9e1007117bcaa199b30d1c.
state=OPENING, ts=1424227700962, server=null*
*
plr_sg_insta_media_live,\x04994537646:78233569a3467:987;7,1410787903804.cd49ec64a0a417aa11949c2bc2d3df6e.
state=OPENING, ts=1424227691774, server=null*
*Issue#2*. The next step that we do is to check HDFS file status using '*hdfs
fsck /*'. It shows that the filesystem '/' is corrupted with these
statistics
* Total size: 15494284950796 B (Total open files size: 17179869184 B)*
* Total dirs: 9198*
* Total files: 124685 (Files currently being written: 21)*
* Total blocks (validated): 219620 (avg. block size 70550427 B) (Total
open file blocks (not validated): 144)*
* *********************************
* CORRUPT FILES: 42*
* MISSING BLOCKS: 142*
* MISSING SIZE: 14899184084 B*
* CORRUPT BLOCKS: 142*
* *********************************
* Corrupt blocks: 142*
* Number of data-nodes: 14*
* Number of racks: 1*
*FSCK ended at Tue Feb 17 17:25:18 SGT 2015 in 3026 milliseconds*
*The filesystem under path '/' is CORRUPT*
So it seems that HDFS loses some of its block due to DN failures and since
the dfs.replication factor is 1, it could not recover the missing blocks.
*Issue#3*. Although '*hbase hbck -repair*' is stuck, we are able to run '*hbase
hbck -fixHdfsHoles*'. We notice this following error messages (I copied
some of them to represent each type of error messages that we have).
- *ERROR: Region { meta =>
plr_id_insta_media_live,\x02:;6;7;398962:3:399a49:653:64,1421565172917.1528f288473632aca2636443574a6ba1.,
hdfs => hdfs://nameservice1/hbase/plr_id_insta_media_live/1528f2884*
*73632aca2636443574a6ba1, deployed => } not deployed on any region server.*
- *ERROR: Region { meta => null, hdfs =>
hdfs://nameservice1/hbase/plr_sg_insta_media_live/8473d25be5980c169bff13cf90229939,
deployed => } on HDFS, but not listed in META or deployed on any region
server*
*- ERROR: Region { meta =>
plr_sg_insta_media_live,\x0293:729769;975376;2a33995622;3,1421985489851.8819ebd296f075513056be4bbd30ee9c.,
hdfs => null, deployed => } found in META, but not in HDFS or deployed on
any region server.*
-ERROR: There is a hole in the region chain between
\x099599464:7:5;3595;8a:57868;95 and \x099;56535:4632439643a82826562:. You
need to create a new .regioninfo and region dir in hdfs to plug the hole.
-ERROR: Last region should end with an empty key. You need to create a new
region and regioninfo in HDFS to plug the hole.
Now to fix this issue, we plan to perform this following action items:
1. Move or delete corrupted files in HDFS
2. Repair HBase by deleting the reference of corrupted files/blocks from
HBase meta tables (it’s okay to lost some of the data)
3. Or create empty HFiles as shown in
http://comments.gmane.org/gmane.comp.java.hadoop.hbase.user/31308
And our questions are:
1. Is it safe to move or delete corrupted files in HDFS? Can we make
HBase to ignore those files and delete corresponding HBase files?
2. Any comments on our action items?
Best regards,
Arinto
www.otnira.com
Re: HBase Region always in transition + corrupt HDFS
Posted by Arinto Murdopo <ar...@gmail.com>.
Hi all
While digging through /hbase folder in HDFS, I found WAL folder that is
splitting and last modified time is 2 weeks ago (I changed the node name in
this example)
*drwxr-xr-x - hbase hbase 0 2015-02-11 18:16
/hbase/.logs/<node2>,60020,1417575124373-splitting*
Is this case common? After reading about log splitting, my understanding is
this folder should disappear after successful recovery. CMIIW.
Best regards,
Arinto
Arinto
www.otnira.com
On Tue, Feb 24, 2015 at 11:45 AM, Arinto Murdopo <ar...@gmail.com> wrote:
>
> On Tue, Feb 24, 2015 at 9:46 AM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
> > I don't have the list of the corrupted files yet. I notice that when I
>> try
>> > to Get some of the files, my HBase client code throws these exceptions:
>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
>> > attempts=2, exceptions:
>> > Mon Feb 23 17:49:32 SGT 2015,
>> > org.apache.hadoop.hbase.client.HTable$3@11ff4a1c,
>> > org.apache.hadoop.hbase.NotServingRegionException:
>> > org.apache.hadoop.hbase.NotServingRegionException: Region is not online:
>> >
>> >
>> plr_sg_insta_media_live,\x0177998597896:953:5:a5:58786,1410771627251.6c323832d2dc77c586f1cf6441c7ef6e.
>> >
>>
>> FSCK should give ou the list of corrupt files. Can you extract it from
>> there?
>>
>
> Yup, I managed to extract them. We have corrupt files as well as missing
> files. Luckily there's no .regionfile corrupted or missing. I'll read more
> about HFile before updating this thread more. :)
>
>
>> >
>> > Can I use these exceptions to determine the corrupted files?
>> > The files are media data (images or videos) obtained from the internet.
>> >
>>
>> This exception gives you all the hints for a directory most probably under
>> /hbase/plr_sg_insta_media_live/6c323832d2dc77c586f1cf6441c7ef6e
>>
>> Files under this directory might be corrupted but you need to find which
>> files. If it's a HFiles it's easy. If it's the .regioninfo it's a bit more
>> tricky.
>
>
>
>
> Arinto
> www.otnira.com
>
Re: HBase Region always in transition + corrupt HDFS
Posted by Arinto Murdopo <ar...@gmail.com>.
On Tue, Feb 24, 2015 at 9:46 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> I don't have the list of the corrupted files yet. I notice that when I try
> > to Get some of the files, my HBase client code throws these exceptions:
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> > attempts=2, exceptions:
> > Mon Feb 23 17:49:32 SGT 2015,
> > org.apache.hadoop.hbase.client.HTable$3@11ff4a1c,
> > org.apache.hadoop.hbase.NotServingRegionException:
> > org.apache.hadoop.hbase.NotServingRegionException: Region is not online:
> >
> >
> plr_sg_insta_media_live,\x0177998597896:953:5:a5:58786,1410771627251.6c323832d2dc77c586f1cf6441c7ef6e.
> >
>
> FSCK should give ou the list of corrupt files. Can you extract it from
> there?
>
Yup, I managed to extract them. We have corrupt files as well as missing
files. Luckily there's no .regionfile corrupted or missing. I'll read more
about HFile before updating this thread more. :)
> >
> > Can I use these exceptions to determine the corrupted files?
> > The files are media data (images or videos) obtained from the internet.
> >
>
> This exception gives you all the hints for a directory most probably under
> /hbase/plr_sg_insta_media_live/6c323832d2dc77c586f1cf6441c7ef6e
>
> Files under this directory might be corrupted but you need to find which
> files. If it's a HFiles it's easy. If it's the .regioninfo it's a bit more
> tricky.
Arinto
www.otnira.com
Re: HBase Region always in transition + corrupt HDFS
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
2015-02-23 20:25 GMT-05:00 Arinto Murdopo <ar...@gmail.com>:
> @JM:
> You mentioned about deleting "the files", are you referring to HDFS files
> or file on HBase?
>
Your HBase files are stored in HDFS. So I think we are refering to the same
thing. Look into /hbase in our HDFS to find HBase files.
>
> Our cluster have 15 nodes. We used 14 of them as DN. Actually we tried to
> enable the remaining one as DN (so that we have 15 DN), but then we
> disabled it (so now we have 14 again). Probably our crawlers write some
> data into the additional DN without any replication. Maybe I could try to
> enable again the DN.
>
That's a very valid option. If you still have the DN directories, just
enable it back to see if you can recover the blocks...
> I don't have the list of the corrupted files yet. I notice that when I try
> to Get some of the files, my HBase client code throws these exceptions:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=2, exceptions:
> Mon Feb 23 17:49:32 SGT 2015,
> org.apache.hadoop.hbase.client.HTable$3@11ff4a1c,
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Region is not online:
>
> plr_sg_insta_media_live,\x0177998597896:953:5:a5:58786,1410771627251.6c323832d2dc77c586f1cf6441c7ef6e.
>
FSCK should give ou the list of corrupt files. Can you extract it from
there?
>
> Can I use these exceptions to determine the corrupted files?
> The files are media data (images or videos) obtained from the internet.
>
This exception gives you all the hints for a directory most probably under
/hbase/plr_sg_insta_media_live/6c323832d2dc77c586f1cf6441c7ef6e
Files under this directory might be corrupted but you need to find which
files. If it's a HFiles it's easy. If it's the .regioninfo it's a bit more
tricky.
JM
> Arinto
> www.otnira.com
>
> On Tue, Feb 24, 2015 at 8:06 AM, Michael Segel <ms...@segel.com> wrote:
>
> > I’m sorry, but I implied checking the checksums of the blocks.
> > Didn’t think I needed to spell it out. Next time I’ll be a bit more
> > precise.
> >
> > > On Feb 23, 2015, at 2:34 PM, Nick Dimiduk <nd...@gmail.com> wrote:
> > >
> > > HBase/HDFS are maintaining block checksums, so presumably a corrupted
> > block
> > > would fail checksum validation. Increasing the number of replicas
> > increases
> > > the odds that you'll still have a valid block. I'm not an HDFS expert,
> > but
> > > I would be very surprised if HDFS is validating a "questionable block"
> > via
> > > byte-wise comparison over the network amongst the replica peers.
> > >
> > > On Mon, Feb 23, 2015 at 12:25 PM, Michael Segel <ms...@segel.com>
> > wrote:
> > >
> > >>
> > >> On Feb 23, 2015, at 1:47 AM, Arinto Murdopo <ar...@gmail.com> wrote:
> > >>
> > >> We're running HBase (0.94.15-cdh4.6.0) on top of HDFS (Hadoop
> > >> 2.0.0-cdh4.6.0).
> > >> For all of our tables, we set the replication factor to 1
> > (dfs.replication
> > >> = 1 in hbase-site.xml). We set to 1 because we want to minimize the
> HDFS
> > >> usage (now we realize we should set this value to at least 2, because
> > >> "failure is a norm" in distributed systems).
> > >>
> > >>
> > >>
> > >> Sorry, but you really want this to be a replication value of at least
> 3
> > >> and not 2.
> > >>
> > >> Suppose you have corruption but not a lost block. Which copy of the
> two
> > is
> > >> right?
> > >> With 3, you can compare the three and hopefully 2 of the 3 will match.
> > >>
> > >>
> >
> >
>
Re: HBase Region always in transition + corrupt HDFS
Posted by Ted Yu <yu...@gmail.com>.
Arinto:
Probably you should take a look at HBASE-12949.
Cheers
On Mon, Feb 23, 2015 at 5:25 PM, Arinto Murdopo <ar...@gmail.com> wrote:
> @JM:
> You mentioned about deleting "the files", are you referring to HDFS files
> or file on HBase?
>
> Our cluster have 15 nodes. We used 14 of them as DN. Actually we tried to
> enable the remaining one as DN (so that we have 15 DN), but then we
> disabled it (so now we have 14 again). Probably our crawlers write some
> data into the additional DN without any replication. Maybe I could try to
> enable again the DN.
>
> I don't have the list of the corrupted files yet. I notice that when I try
> to Get some of the files, my HBase client code throws these exceptions:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=2, exceptions:
> Mon Feb 23 17:49:32 SGT 2015,
> org.apache.hadoop.hbase.client.HTable$3@11ff4a1c,
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Region is not online:
>
> plr_sg_insta_media_live,\x0177998597896:953:5:a5:58786,1410771627251.6c323832d2dc77c586f1cf6441c7ef6e.
>
> Can I use these exceptions to determine the corrupted files?
> The files are media data (images or videos) obtained from the internet.
>
> @Michael Segel: Yup, 3 is the default and recommended value. We were
> overwhelmed with the amount of data, so we foolishly reduced our
> replication factor. We have learnt the lesson the hard way :).
>
> Fortunately it's okay to lose the data, i.e. we can easily recover them
> from our other data.
>
>
>
> Arinto
> www.otnira.com
>
> On Tue, Feb 24, 2015 at 8:06 AM, Michael Segel <ms...@segel.com> wrote:
>
> > I’m sorry, but I implied checking the checksums of the blocks.
> > Didn’t think I needed to spell it out. Next time I’ll be a bit more
> > precise.
> >
> > > On Feb 23, 2015, at 2:34 PM, Nick Dimiduk <nd...@gmail.com> wrote:
> > >
> > > HBase/HDFS are maintaining block checksums, so presumably a corrupted
> > block
> > > would fail checksum validation. Increasing the number of replicas
> > increases
> > > the odds that you'll still have a valid block. I'm not an HDFS expert,
> > but
> > > I would be very surprised if HDFS is validating a "questionable block"
> > via
> > > byte-wise comparison over the network amongst the replica peers.
> > >
> > > On Mon, Feb 23, 2015 at 12:25 PM, Michael Segel <ms...@segel.com>
> > wrote:
> > >
> > >>
> > >> On Feb 23, 2015, at 1:47 AM, Arinto Murdopo <ar...@gmail.com> wrote:
> > >>
> > >> We're running HBase (0.94.15-cdh4.6.0) on top of HDFS (Hadoop
> > >> 2.0.0-cdh4.6.0).
> > >> For all of our tables, we set the replication factor to 1
> > (dfs.replication
> > >> = 1 in hbase-site.xml). We set to 1 because we want to minimize the
> HDFS
> > >> usage (now we realize we should set this value to at least 2, because
> > >> "failure is a norm" in distributed systems).
> > >>
> > >>
> > >>
> > >> Sorry, but you really want this to be a replication value of at least
> 3
> > >> and not 2.
> > >>
> > >> Suppose you have corruption but not a lost block. Which copy of the
> two
> > is
> > >> right?
> > >> With 3, you can compare the three and hopefully 2 of the 3 will match.
> > >>
> > >>
> >
> >
>
Re: HBase Region always in transition + corrupt HDFS
Posted by Arinto Murdopo <ar...@gmail.com>.
@JM:
You mentioned about deleting "the files", are you referring to HDFS files
or file on HBase?
Our cluster have 15 nodes. We used 14 of them as DN. Actually we tried to
enable the remaining one as DN (so that we have 15 DN), but then we
disabled it (so now we have 14 again). Probably our crawlers write some
data into the additional DN without any replication. Maybe I could try to
enable again the DN.
I don't have the list of the corrupted files yet. I notice that when I try
to Get some of the files, my HBase client code throws these exceptions:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=2, exceptions:
Mon Feb 23 17:49:32 SGT 2015,
org.apache.hadoop.hbase.client.HTable$3@11ff4a1c,
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Region is not online:
plr_sg_insta_media_live,\x0177998597896:953:5:a5:58786,1410771627251.6c323832d2dc77c586f1cf6441c7ef6e.
Can I use these exceptions to determine the corrupted files?
The files are media data (images or videos) obtained from the internet.
@Michael Segel: Yup, 3 is the default and recommended value. We were
overwhelmed with the amount of data, so we foolishly reduced our
replication factor. We have learnt the lesson the hard way :).
Fortunately it's okay to lose the data, i.e. we can easily recover them
from our other data.
Arinto
www.otnira.com
On Tue, Feb 24, 2015 at 8:06 AM, Michael Segel <ms...@segel.com> wrote:
> I’m sorry, but I implied checking the checksums of the blocks.
> Didn’t think I needed to spell it out. Next time I’ll be a bit more
> precise.
>
> > On Feb 23, 2015, at 2:34 PM, Nick Dimiduk <nd...@gmail.com> wrote:
> >
> > HBase/HDFS are maintaining block checksums, so presumably a corrupted
> block
> > would fail checksum validation. Increasing the number of replicas
> increases
> > the odds that you'll still have a valid block. I'm not an HDFS expert,
> but
> > I would be very surprised if HDFS is validating a "questionable block"
> via
> > byte-wise comparison over the network amongst the replica peers.
> >
> > On Mon, Feb 23, 2015 at 12:25 PM, Michael Segel <ms...@segel.com>
> wrote:
> >
> >>
> >> On Feb 23, 2015, at 1:47 AM, Arinto Murdopo <ar...@gmail.com> wrote:
> >>
> >> We're running HBase (0.94.15-cdh4.6.0) on top of HDFS (Hadoop
> >> 2.0.0-cdh4.6.0).
> >> For all of our tables, we set the replication factor to 1
> (dfs.replication
> >> = 1 in hbase-site.xml). We set to 1 because we want to minimize the HDFS
> >> usage (now we realize we should set this value to at least 2, because
> >> "failure is a norm" in distributed systems).
> >>
> >>
> >>
> >> Sorry, but you really want this to be a replication value of at least 3
> >> and not 2.
> >>
> >> Suppose you have corruption but not a lost block. Which copy of the two
> is
> >> right?
> >> With 3, you can compare the three and hopefully 2 of the 3 will match.
> >>
> >>
>
>
Re: HBase Region always in transition + corrupt HDFS
Posted by Michael Segel <ms...@segel.com>.
I’m sorry, but I implied checking the checksums of the blocks.
Didn’t think I needed to spell it out. Next time I’ll be a bit more precise.
> On Feb 23, 2015, at 2:34 PM, Nick Dimiduk <nd...@gmail.com> wrote:
>
> HBase/HDFS are maintaining block checksums, so presumably a corrupted block
> would fail checksum validation. Increasing the number of replicas increases
> the odds that you'll still have a valid block. I'm not an HDFS expert, but
> I would be very surprised if HDFS is validating a "questionable block" via
> byte-wise comparison over the network amongst the replica peers.
>
> On Mon, Feb 23, 2015 at 12:25 PM, Michael Segel <ms...@segel.com> wrote:
>
>>
>> On Feb 23, 2015, at 1:47 AM, Arinto Murdopo <ar...@gmail.com> wrote:
>>
>> We're running HBase (0.94.15-cdh4.6.0) on top of HDFS (Hadoop
>> 2.0.0-cdh4.6.0).
>> For all of our tables, we set the replication factor to 1 (dfs.replication
>> = 1 in hbase-site.xml). We set to 1 because we want to minimize the HDFS
>> usage (now we realize we should set this value to at least 2, because
>> "failure is a norm" in distributed systems).
>>
>>
>>
>> Sorry, but you really want this to be a replication value of at least 3
>> and not 2.
>>
>> Suppose you have corruption but not a lost block. Which copy of the two is
>> right?
>> With 3, you can compare the three and hopefully 2 of the 3 will match.
>>
>>
Re: HBase Region always in transition + corrupt HDFS
Posted by Nick Dimiduk <nd...@gmail.com>.
HBase/HDFS are maintaining block checksums, so presumably a corrupted block
would fail checksum validation. Increasing the number of replicas increases
the odds that you'll still have a valid block. I'm not an HDFS expert, but
I would be very surprised if HDFS is validating a "questionable block" via
byte-wise comparison over the network amongst the replica peers.
On Mon, Feb 23, 2015 at 12:25 PM, Michael Segel <ms...@segel.com> wrote:
>
> On Feb 23, 2015, at 1:47 AM, Arinto Murdopo <ar...@gmail.com> wrote:
>
> We're running HBase (0.94.15-cdh4.6.0) on top of HDFS (Hadoop
> 2.0.0-cdh4.6.0).
> For all of our tables, we set the replication factor to 1 (dfs.replication
> = 1 in hbase-site.xml). We set to 1 because we want to minimize the HDFS
> usage (now we realize we should set this value to at least 2, because
> "failure is a norm" in distributed systems).
>
>
>
> Sorry, but you really want this to be a replication value of at least 3
> and not 2.
>
> Suppose you have corruption but not a lost block. Which copy of the two is
> right?
> With 3, you can compare the three and hopefully 2 of the 3 will match.
>
>
Re: HBase Region always in transition + corrupt HDFS
Posted by Michael Segel <ms...@segel.com>.
> On Feb 23, 2015, at 1:47 AM, Arinto Murdopo <ar...@gmail.com> wrote:
>
> We're running HBase (0.94.15-cdh4.6.0) on top of HDFS (Hadoop
> 2.0.0-cdh4.6.0).
> For all of our tables, we set the replication factor to 1 (dfs.replication
> = 1 in hbase-site.xml). We set to 1 because we want to minimize the HDFS
> usage (now we realize we should set this value to at least 2, because
> "failure is a norm" in distributed systems).
Sorry, but you really want this to be a replication value of at least 3 and not 2.
Suppose you have corruption but not a lost block. Which copy of the two is right?
With 3, you can compare the three and hopefully 2 of the 3 will match.
Re: HBase Region always in transition + corrupt HDFS
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
You have no other choice than removing those files... you will loose the
related data but it should be fine if they are only HFiles. Do you have the
list of corrupted files? What kind of files it is?
Also, have you lost a node or a disk? How have you lost about 150 blocks?
JM
2015-02-23 2:47 GMT-05:00 Arinto Murdopo <ar...@gmail.com>:
> Hi all,
>
> We're running HBase (0.94.15-cdh4.6.0) on top of HDFS (Hadoop
> 2.0.0-cdh4.6.0).
> For all of our tables, we set the replication factor to 1 (dfs.replication
> = 1 in hbase-site.xml). We set to 1 because we want to minimize the HDFS
> usage (now we realize we should set this value to at least 2, because
> "failure is a norm" in distributed systems).
>
> Due to the amount of data, at some point, we have low disk space in HDFS
> and one of our DNs was down. Now we have these problems in HBase and HDFS
> although we have recovered our DN.
>
> *Issue#1*. Some of HBase region always in transition. '*hbase hbck
> -repair*'
> is stuck because it's waiting for region transition to finish. Some output
>
> *hbase(main):003:0> status 'detailed'*
> *12 regionsInTransition*
> *
>
> plr_id_insta_media_live,\x02:;6;7;398962:3:399a49:653:64,1421565172917.1528f288473632aca2636443574a6ba1.
> state=OPENING, ts=1424227696897, server=null*
> *
>
> plr_sg_insta_media_live,\x0098;522:997;8798665a64;67879,1410768824800.2c79bbc5c0dc2d2b39c04c8abc0a90ff.
> state=OFFLINE, ts=1424227714203, server=null*
> *
>
> plr_sg_insta_media_live,\x00465892:9935773828;a4459;649,1410767723471.55097cfc60bc9f50303dadb02abcd64b.
> state=OPENING, ts=1424227701234, server=null*
> *
>
> plr_sg_insta_media_live,\x00474973488232837733a38744,1410767723471.740d6655afb74a2ff421c6ef16037f57.
> state=OPENING, ts=1424227708053, server=null*
> *
>
> plr_id_insta_media_live,\x02::449::4;:466;3988a6432677;3,1419435100617.7caf3d749dce37037eec9ccc29d272a1.
> state=OPENING, ts=1424227701484, server=null*
> *
>
> plr_sg_insta_media_live,\x05779793546323;::4:4a3:8227928,1418845792479.81c4da129ae5b7b204d5373d9e0fea3d.
> state=OPENING, ts=1424227705353, server=null*
> *
>
> plr_sg_insta_media_live,\x009;5:686348963:33:5a5634887,1410769837567.8a9ded24960a7787ca016e2073b24151.
> state=OPENING, ts=1424227706293, server=null*
> *
>
> plr_sg_insta_media_live,\x0375;6;7377578;84226a7663792,1418980694076.a1e1c98f646ee899010f19a9c693c67c.
> state=OPENING, ts=1424227680569, server=null*
> *
>
> plr_sg_insta_media_live,\x018;3826368274679364a3;;73457;,1421425643816.b04ffda1b2024bac09c9e6246fb7b183.
> state=OPENING, ts=1424227680538, server=null*
> *
>
> plr_sg_insta_media_live,\x0154752;22:43377542:a:86:239,1410771044924.c57d6b4d23f21d3e914a91721a99ce12.
> state=OPENING, ts=1424227710847, server=null*
> *
>
> plr_sg_insta_media_live,\x0069;7;9384697:;8685a885485:,1410767928822.c7b5e53cdd9e1007117bcaa199b30d1c.
> state=OPENING, ts=1424227700962, server=null*
> *
>
> plr_sg_insta_media_live,\x04994537646:78233569a3467:987;7,1410787903804.cd49ec64a0a417aa11949c2bc2d3df6e.
> state=OPENING, ts=1424227691774, server=null*
>
>
> *Issue#2*. The next step that we do is to check HDFS file status using
> '*hdfs
> fsck /*'. It shows that the filesystem '/' is corrupted with these
> statistics
> * Total size: 15494284950796 B (Total open files size: 17179869184 B)*
> * Total dirs: 9198*
> * Total files: 124685 (Files currently being written: 21)*
> * Total blocks (validated): 219620 (avg. block size 70550427 B) (Total
> open file blocks (not validated): 144)*
> * *********************************
> * CORRUPT FILES: 42*
> * MISSING BLOCKS: 142*
> * MISSING SIZE: 14899184084 B*
> * CORRUPT BLOCKS: 142*
> * *********************************
> * Corrupt blocks: 142*
> * Number of data-nodes: 14*
> * Number of racks: 1*
> *FSCK ended at Tue Feb 17 17:25:18 SGT 2015 in 3026 milliseconds*
>
>
> *The filesystem under path '/' is CORRUPT*
>
> So it seems that HDFS loses some of its block due to DN failures and since
> the dfs.replication factor is 1, it could not recover the missing blocks.
>
> *Issue#3*. Although '*hbase hbck -repair*' is stuck, we are able to run
> '*hbase
> hbck -fixHdfsHoles*'. We notice this following error messages (I copied
> some of them to represent each type of error messages that we have).
> - *ERROR: Region { meta =>
>
> plr_id_insta_media_live,\x02:;6;7;398962:3:399a49:653:64,1421565172917.1528f288473632aca2636443574a6ba1.,
> hdfs => hdfs://nameservice1/hbase/plr_id_insta_media_live/1528f2884*
> *73632aca2636443574a6ba1, deployed => } not deployed on any region
> server.*
> - *ERROR: Region { meta => null, hdfs =>
>
> hdfs://nameservice1/hbase/plr_sg_insta_media_live/8473d25be5980c169bff13cf90229939,
> deployed => } on HDFS, but not listed in META or deployed on any region
> server*
> *- ERROR: Region { meta =>
>
> plr_sg_insta_media_live,\x0293:729769;975376;2a33995622;3,1421985489851.8819ebd296f075513056be4bbd30ee9c.,
> hdfs => null, deployed => } found in META, but not in HDFS or deployed on
> any region server.*
> -ERROR: There is a hole in the region chain between
> \x099599464:7:5;3595;8a:57868;95 and \x099;56535:4632439643a82826562:. You
> need to create a new .regioninfo and region dir in hdfs to plug the hole.
> -ERROR: Last region should end with an empty key. You need to create a new
> region and regioninfo in HDFS to plug the hole.
>
> Now to fix this issue, we plan to perform this following action items:
>
> 1. Move or delete corrupted files in HDFS
> 2. Repair HBase by deleting the reference of corrupted files/blocks from
> HBase meta tables (it’s okay to lost some of the data)
> 3. Or create empty HFiles as shown in
> http://comments.gmane.org/gmane.comp.java.hadoop.hbase.user/31308
>
>
> And our questions are:
>
> 1. Is it safe to move or delete corrupted files in HDFS? Can we make
> HBase to ignore those files and delete corresponding HBase files?
> 2. Any comments on our action items?
>
>
> Best regards,
>
> Arinto
> www.otnira.com
>