You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Ted Yu <yu...@gmail.com> on 2013/03/02 16:47:29 UTC
Re: HRegionInfo was null or empty in Meta errors
HBASE-5837 is only in 0.95 and later.
Do you want HBASE-5837 to be backported ?
Thanks
On Thu, Feb 7, 2013 at 3:17 PM, Brandon Peskin <bp...@adobe.com> wrote:
> Thanks Kevin.
>
> Before I tried your advice, I tried this:
>
> scan '.META.', { FILTER =>
> org.apache.hadoop.hbase.filter.SingleColumnValueFilter.new
> (org.apache.hadoop.hbase.util.Bytes.toBytes('info'),
> org.apache.hadoop.hbase.util.Bytes.toBytes('regioninfo'),
>
> org.apache.hadoop.hbase.filter.CompareFilter::CompareOp.valueOf('NOT_EQUAL'),
> org.apache.hadoop.hbase.filter.SubstringComparator.new('algol'))}
>
> deleteall '.META.', '<row_key>'
>
>
> The problem is at some point I fat-fingereda row key and believe I hit
> HBASE-5837
>
> https://issues.apache.org/jira/browse/HBASE-5837
>
> I'm getting java.io.IOException: java.io.IOException:
> java.lang.IllegalArgumentException: No 44 in
> <2e40c841-af5b-4a5e-be0f-e06a953f05cc,1359958540596>, length=13, offset=37
> Caused my master to die, can't restart it.
>
> Is there any way around this or have I completely hosed my hbase
> installation?
>
>
> On Jan 31, 2013, at 6:23 AM, Kevin O'dell <ke...@cloudera.com>
> wrote:
>
> > I am going to disagree with ignoring the error. You will encounter
> > failures when doing other operations such as import/exports. The first
> > thing I would do is like JM said, lets focus on the region that is not in
> > META(we at least want 0 inconsistencies). Can you please run hbck
> -repair
> > and then run another -details and let us know if you are still seeing
> > errors? After that, if you are still getting the NULL errors for
> > hregion:info in META. Can you please run echo "scan '.META.'" | hbase
> shell
> >> meta.out and attach the meta.out file. I would like to take a look at
> > some of these.
> >
> > To be able to run the -repair we will want to use a different jar and
> some
> > instructions:
> >
> >
> > 1. Move the new uber jar on to the system.
> > hbase-0.90.4-cdh3u3-patch30+3.jar
> > 2. Copy the hbase dir(/usr/lib/hbase) into /tmp/hbase dir.
> > 3. Move the hbase jar(hbase-0.90.4-cdh3u2.jar) to a .old from the tmp
> > and replace it with the uber-hbck(hbase-0.90.4-cdh3u3-patch30+3.jar).
> > 4. break the sym links that directory
> > 5. Add the value of fs.default.name from the core-site.xml to the
> > HBase-site.xml
> > 6. export HBASE_HOME=/tmp/hbase/ and run ./bin/hbase hbck -details 2>&1
> > | tee details.out.
> > 7. Check the details.out and make sure you are still seeing
> > inconsistencies
> > 8. ./bin/hbase hbck -repair 2>&1 | tee repair.out.
> > 9. Run -details again and make sure we have 0 inconsistencies.
> >
> >
> >
> https://www.dropbox.com/s/fxotosglrrl1tq2/hbase-0.90.4-cdh3u3-patch30%2B3.jar
> > <--- new jar
> >
> > On Thu, Jan 31, 2013 at 6:48 AM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >>
> >> Hi Brandon,
> >>
> >> I faced the same issue for "HRegionInfo was null or empty" on January
> >> 24th and Ted replied:
> >>
> >> "Encountered problems when prefetch META table:
> >>
> >> You can ignore the warning."
> >>
> >> So I think you should focus on the last one "not listed in META or
> >> deployed on any region server".
> >>
> >> Have you tried hbck to see if it can fix it?
> >>
> >> JM
> >>
> >> 2013/1/31, Brandon Peskin <bp...@adobe.com>:
> >>> hadoop 0.20.2-cdh3u2
> >>> hbase 0.90.4-cdh3u2
> >>>
> >>> On January 8th I had a network event where I lost three region servers.
> >>>
> >>> When they came back I had unassigned regions/regions not being served
> > errors
> >>> which I fixed with the hbck -fix
> >>>
> >>>
> >>>
> >>> Since then, however I have been getting an increasing number of these
> > when I
> >>> have clients trying to write to specific tables:
> >>>
> >>>
> >>> java.io.IOException: HRegionInfo was null or empty in Meta for
> >>> algol_profile_training_record,
> >>>
> >
> row=algol_profile_training_record,clientcode:49128:abce6d9f-1ee2-434a-8a82-a151b7dc183f,99999999999999
> >>> at
> >>>
> > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:142)
> >>> ~[hbase-0.90.4-cdh3u2.jar:na]
> >>> at
> > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:95)
> >>> ~[hbase-0.90.4-cdh3u2.jar:na]
> >>> at
> >>>
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:649)
> >>> [hbase-0.90.4-cdh3u2.jar:na]
> >>>
> >>>
> >>>
> >>> ....to the point now where I seemingly can't even write to that table.
> >>>
> >>> This also coincides with the following, seen in the the hbase master
> > log:
> >>>
> >>> 13/01/31 02:49:55 WARN master.CatalogJanitor: REGIONINFO_QUALIFIER is
> > empty
> >>> in
> >>>
> >
> keyvalues={algol_tmp_client_39691_20121101053400_6,,1351783324210.b703b203d6189cad224764853cbd88f7./info:server/1351785275123/Put/vlen=32,
> >>>
> >
> algol_tmp_clientcode_39691_20121101053400_6,,1351783324210.b703b203d6189cad224764853cbd88f7./info:serverstartcode/1351785275123/Put/vlen=8}
> >>> 13/01/31 02:49:55 WARN master.CatalogJanitor: REGIONINFO_QUALIFIER is
> > empty
> >>> in
> >>>
> >
> keyvalues={algol_tmp_client_39691_20121101053400_6,\x7F\xD6\xA5Pm\xF2E\x9D\x81\xB1b\xD1'\xD3\xE5\xA4,1351783324210.507bdf342aaae45644d597c272c52e02./info:server/1351785275128/Put/vlen=32,
> >>>
> >
> algol_tmp_clientcode_39691_20121101053400_6,\x7F\xD6\xA5Pm\xF2E\x9D\x81\xB1b\xD1'\xD3\xE5\xA4,1351783324210.507bdf342aaae45644d597c272c52e02./info:serverstartcode/1351785275128/Put/vlen=8}
> >>> 13/01/31 02:49:55 WARN master.CatalogJanitor: REGIONINFO_QUALIFIER is
> > empty
> >>> in
> >>>
> >
> keyvalues={algol_tmp_client_34785_20121004102759_0,,1349383219155.a45774fa4c8af765df87d3373d30cfdc./info:server/1349461385375/Put/vlen=32,
> >>>
> >
> algol_tmp_overstock_34785_20121004102759_0,,1349383219155.a45774fa4c8af765df87d3373d30cfdc./info:serverstartcode/1349461385375/Put/vlen=8}
> >>>
> >>>
> >>> ...and there's 58 of these I cannot fix (presumably I could with the
> > hbck
> >>> provided with CDH4):
> >>>
> >>> ERROR: Region
> >>>
> >
> hdfs://namenode:9000/hbase/algol_profile_training_record/fdaa1024d1b725b4997d2283640f0fa4
> >>> on HDFS, but not listed in META or deployed on any region server
> >>>
> >>>
> >>> I'm absolutely stumped. I've done some poking around and I can't find
> > any
> >>> sort of data surrounding this issue with the exception of similar
> > symptoms
> >>> in an exchange on this list of March this year (though the inquirer had
> >>> different questions). Any help would be appreciated, though I suspect I
> > will
> >>> be told 'Upgrade to CDH4' or 'Drop and re-create the table'.
> >>>
> >>> Thanks in advance.
> >>>
> >>> --
> >>> Brandon Peskin
> >>> Senior Systems Administrator
> >>> Adobe Systems
> >>> bpeskin@adobe.com
> >>>
> >>>
> >>>
> >>>
> >>>
> >
> >
> >
> >
> > --
> > Kevin O'Dell
> > Customer Operations Engineer, Cloudera
>
>
Re: HRegionInfo was null or empty in Meta errors
Posted by Ted Yu <yu...@gmail.com>.
I have created HBASE-7990 for backport.
On Mon, Mar 4, 2013 at 4:28 AM, Ted <yu...@gmail.com> wrote:
> We should back port.
>
> Thanks
>
> On Mar 4, 2013, at 4:16 AM, "Kevin O'dell" <ke...@cloudera.com>
> wrote:
>
> > Ted,
> >
> > This is the second or third time I have seen this. I think it should
> > apply fairly clean. What do you think?
> >
> > On Sat, Mar 2, 2013 at 10:47 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> >> HBASE-5837 is only in 0.95 and later.
> >>
> >> Do you want HBASE-5837 to be backported ?
> >>
> >> Thanks
> >>
> >> On Thu, Feb 7, 2013 at 3:17 PM, Brandon Peskin <bp...@adobe.com>
> wrote:
> >>
> >>> Thanks Kevin.
> >>>
> >>> Before I tried your advice, I tried this:
> >>>
> >>> scan '.META.', { FILTER =>
> >>> org.apache.hadoop.hbase.filter.SingleColumnValueFilter.new
> >>> (org.apache.hadoop.hbase.util.Bytes.toBytes('info'),
> >>> org.apache.hadoop.hbase.util.Bytes.toBytes('regioninfo'),
> >>
> org.apache.hadoop.hbase.filter.CompareFilter::CompareOp.valueOf('NOT_EQUAL'),
> >>> org.apache.hadoop.hbase.filter.SubstringComparator.new('algol'))}
> >>>
> >>> deleteall '.META.', '<row_key>'
> >>>
> >>>
> >>> The problem is at some point I fat-fingereda row key and believe I hit
> >>> HBASE-5837
> >>>
> >>> https://issues.apache.org/jira/browse/HBASE-5837
> >>>
> >>> I'm getting java.io.IOException: java.io.IOException:
> >>> java.lang.IllegalArgumentException: No 44 in
> >>> <2e40c841-af5b-4a5e-be0f-e06a953f05cc,1359958540596>, length=13,
> >> offset=37
> >>> Caused my master to die, can't restart it.
> >>>
> >>> Is there any way around this or have I completely hosed my hbase
> >>> installation?
> >>>
> >>>
> >>> On Jan 31, 2013, at 6:23 AM, Kevin O'dell <ke...@cloudera.com>
> >>> wrote:
> >>>
> >>>> I am going to disagree with ignoring the error. You will encounter
> >>>> failures when doing other operations such as import/exports. The
> first
> >>>> thing I would do is like JM said, lets focus on the region that is not
> >> in
> >>>> META(we at least want 0 inconsistencies). Can you please run hbck
> >>> -repair
> >>>> and then run another -details and let us know if you are still seeing
> >>>> errors? After that, if you are still getting the NULL errors for
> >>>> hregion:info in META. Can you please run echo "scan '.META.'" | hbase
> >>> shell
> >>>>> meta.out and attach the meta.out file. I would like to take a look
> at
> >>>> some of these.
> >>>>
> >>>> To be able to run the -repair we will want to use a different jar and
> >>> some
> >>>> instructions:
> >>>>
> >>>>
> >>>> 1. Move the new uber jar on to the system.
> >>>> hbase-0.90.4-cdh3u3-patch30+3.jar
> >>>> 2. Copy the hbase dir(/usr/lib/hbase) into /tmp/hbase dir.
> >>>> 3. Move the hbase jar(hbase-0.90.4-cdh3u2.jar) to a .old from the tmp
> >>>> and replace it with the uber-hbck(hbase-0.90.4-cdh3u3-patch30+3.jar).
> >>>> 4. break the sym links that directory
> >>>> 5. Add the value of fs.default.name from the core-site.xml to the
> >>>> HBase-site.xml
> >>>> 6. export HBASE_HOME=/tmp/hbase/ and run ./bin/hbase hbck -details
> >> 2>&1
> >>>> | tee details.out.
> >>>> 7. Check the details.out and make sure you are still seeing
> >>>> inconsistencies
> >>>> 8. ./bin/hbase hbck -repair 2>&1 | tee repair.out.
> >>>> 9. Run -details again and make sure we have 0 inconsistencies.
> >>
> https://www.dropbox.com/s/fxotosglrrl1tq2/hbase-0.90.4-cdh3u3-patch30%2B3.jar
> >>>> <--- new jar
> >>>>
> >>>> On Thu, Jan 31, 2013 at 6:48 AM, Jean-Marc Spaggiari <
> >>>> jean-marc@spaggiari.org> wrote:
> >>>>>
> >>>>> Hi Brandon,
> >>>>>
> >>>>> I faced the same issue for "HRegionInfo was null or empty" on January
> >>>>> 24th and Ted replied:
> >>>>>
> >>>>> "Encountered problems when prefetch META table:
> >>>>>
> >>>>> You can ignore the warning."
> >>>>>
> >>>>> So I think you should focus on the last one "not listed in META or
> >>>>> deployed on any region server".
> >>>>>
> >>>>> Have you tried hbck to see if it can fix it?
> >>>>>
> >>>>> JM
> >>>>>
> >>>>> 2013/1/31, Brandon Peskin <bp...@adobe.com>:
> >>>>>> hadoop 0.20.2-cdh3u2
> >>>>>> hbase 0.90.4-cdh3u2
> >>>>>>
> >>>>>> On January 8th I had a network event where I lost three region
> >> servers.
> >>>>>>
> >>>>>> When they came back I had unassigned regions/regions not being
> served
> >>>> errors
> >>>>>> which I fixed with the hbck -fix
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Since then, however I have been getting an increasing number of
> these
> >>>> when I
> >>>>>> have clients trying to write to specific tables:
> >>>>>>
> >>>>>>
> >>>>>> java.io.IOException: HRegionInfo was null or empty in Meta for
> >>>>>> algol_profile_training_record,
> >>
> row=algol_profile_training_record,clientcode:49128:abce6d9f-1ee2-434a-8a82-a151b7dc183f,99999999999999
> >>>>>> at
> >>
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:142)
> >>>>>> ~[hbase-0.90.4-cdh3u2.jar:na]
> >>>>>> at
> >> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:95)
> >>>>>> ~[hbase-0.90.4-cdh3u2.jar:na]
> >>>>>> at
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:649)
> >>>>>> [hbase-0.90.4-cdh3u2.jar:na]
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ....to the point now where I seemingly can't even write to that
> >> table.
> >>>>>>
> >>>>>> This also coincides with the following, seen in the the hbase master
> >>>> log:
> >>>>>>
> >>>>>> 13/01/31 02:49:55 WARN master.CatalogJanitor: REGIONINFO_QUALIFIER
> is
> >>>> empty
> >>>>>> in
> >>
> keyvalues={algol_tmp_client_39691_20121101053400_6,,1351783324210.b703b203d6189cad224764853cbd88f7./info:server/1351785275123/Put/vlen=32,
> >>
> algol_tmp_clientcode_39691_20121101053400_6,,1351783324210.b703b203d6189cad224764853cbd88f7./info:serverstartcode/1351785275123/Put/vlen=8}
> >>>>>> 13/01/31 02:49:55 WARN master.CatalogJanitor: REGIONINFO_QUALIFIER
> is
> >>>> empty
> >>>>>> in
> >>
> keyvalues={algol_tmp_client_39691_20121101053400_6,\x7F\xD6\xA5Pm\xF2E\x9D\x81\xB1b\xD1'\xD3\xE5\xA4,1351783324210.507bdf342aaae45644d597c272c52e02./info:server/1351785275128/Put/vlen=32,
> >>
> algol_tmp_clientcode_39691_20121101053400_6,\x7F\xD6\xA5Pm\xF2E\x9D\x81\xB1b\xD1'\xD3\xE5\xA4,1351783324210.507bdf342aaae45644d597c272c52e02./info:serverstartcode/1351785275128/Put/vlen=8}
> >>>>>> 13/01/31 02:49:55 WARN master.CatalogJanitor: REGIONINFO_QUALIFIER
> is
> >>>> empty
> >>>>>> in
> >>
> keyvalues={algol_tmp_client_34785_20121004102759_0,,1349383219155.a45774fa4c8af765df87d3373d30cfdc./info:server/1349461385375/Put/vlen=32,
> >>
> algol_tmp_overstock_34785_20121004102759_0,,1349383219155.a45774fa4c8af765df87d3373d30cfdc./info:serverstartcode/1349461385375/Put/vlen=8}
> >>>>>>
> >>>>>>
> >>>>>> ...and there's 58 of these I cannot fix (presumably I could with the
> >>>> hbck
> >>>>>> provided with CDH4):
> >>>>>>
> >>>>>> ERROR: Region
> >>
> hdfs://namenode:9000/hbase/algol_profile_training_record/fdaa1024d1b725b4997d2283640f0fa4
> >>>>>> on HDFS, but not listed in META or deployed on any region server
> >>>>>>
> >>>>>>
> >>>>>> I'm absolutely stumped. I've done some poking around and I can't
> find
> >>>> any
> >>>>>> sort of data surrounding this issue with the exception of similar
> >>>> symptoms
> >>>>>> in an exchange on this list of March this year (though the inquirer
> >> had
> >>>>>> different questions). Any help would be appreciated, though I
> >> suspect I
> >>>> will
> >>>>>> be told 'Upgrade to CDH4' or 'Drop and re-create the table'.
> >>>>>>
> >>>>>> Thanks in advance.
> >>>>>>
> >>>>>> --
> >>>>>> Brandon Peskin
> >>>>>> Senior Systems Administrator
> >>>>>> Adobe Systems
> >>>>>> bpeskin@adobe.com
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Kevin O'Dell
> >>>> Customer Operations Engineer, Cloudera
> >
> >
> >
> > --
> > Kevin O'Dell
> > Customer Operations Engineer, Cloudera
>
Re: HRegionInfo was null or empty in Meta errors
Posted by Ted <yu...@gmail.com>.
We should back port.
Thanks
On Mar 4, 2013, at 4:16 AM, "Kevin O'dell" <ke...@cloudera.com> wrote:
> Ted,
>
> This is the second or third time I have seen this. I think it should
> apply fairly clean. What do you think?
>
> On Sat, Mar 2, 2013 at 10:47 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> HBASE-5837 is only in 0.95 and later.
>>
>> Do you want HBASE-5837 to be backported ?
>>
>> Thanks
>>
>> On Thu, Feb 7, 2013 at 3:17 PM, Brandon Peskin <bp...@adobe.com> wrote:
>>
>>> Thanks Kevin.
>>>
>>> Before I tried your advice, I tried this:
>>>
>>> scan '.META.', { FILTER =>
>>> org.apache.hadoop.hbase.filter.SingleColumnValueFilter.new
>>> (org.apache.hadoop.hbase.util.Bytes.toBytes('info'),
>>> org.apache.hadoop.hbase.util.Bytes.toBytes('regioninfo'),
>> org.apache.hadoop.hbase.filter.CompareFilter::CompareOp.valueOf('NOT_EQUAL'),
>>> org.apache.hadoop.hbase.filter.SubstringComparator.new('algol'))}
>>>
>>> deleteall '.META.', '<row_key>'
>>>
>>>
>>> The problem is at some point I fat-fingereda row key and believe I hit
>>> HBASE-5837
>>>
>>> https://issues.apache.org/jira/browse/HBASE-5837
>>>
>>> I'm getting java.io.IOException: java.io.IOException:
>>> java.lang.IllegalArgumentException: No 44 in
>>> <2e40c841-af5b-4a5e-be0f-e06a953f05cc,1359958540596>, length=13,
>> offset=37
>>> Caused my master to die, can't restart it.
>>>
>>> Is there any way around this or have I completely hosed my hbase
>>> installation?
>>>
>>>
>>> On Jan 31, 2013, at 6:23 AM, Kevin O'dell <ke...@cloudera.com>
>>> wrote:
>>>
>>>> I am going to disagree with ignoring the error. You will encounter
>>>> failures when doing other operations such as import/exports. The first
>>>> thing I would do is like JM said, lets focus on the region that is not
>> in
>>>> META(we at least want 0 inconsistencies). Can you please run hbck
>>> -repair
>>>> and then run another -details and let us know if you are still seeing
>>>> errors? After that, if you are still getting the NULL errors for
>>>> hregion:info in META. Can you please run echo "scan '.META.'" | hbase
>>> shell
>>>>> meta.out and attach the meta.out file. I would like to take a look at
>>>> some of these.
>>>>
>>>> To be able to run the -repair we will want to use a different jar and
>>> some
>>>> instructions:
>>>>
>>>>
>>>> 1. Move the new uber jar on to the system.
>>>> hbase-0.90.4-cdh3u3-patch30+3.jar
>>>> 2. Copy the hbase dir(/usr/lib/hbase) into /tmp/hbase dir.
>>>> 3. Move the hbase jar(hbase-0.90.4-cdh3u2.jar) to a .old from the tmp
>>>> and replace it with the uber-hbck(hbase-0.90.4-cdh3u3-patch30+3.jar).
>>>> 4. break the sym links that directory
>>>> 5. Add the value of fs.default.name from the core-site.xml to the
>>>> HBase-site.xml
>>>> 6. export HBASE_HOME=/tmp/hbase/ and run ./bin/hbase hbck -details
>> 2>&1
>>>> | tee details.out.
>>>> 7. Check the details.out and make sure you are still seeing
>>>> inconsistencies
>>>> 8. ./bin/hbase hbck -repair 2>&1 | tee repair.out.
>>>> 9. Run -details again and make sure we have 0 inconsistencies.
>> https://www.dropbox.com/s/fxotosglrrl1tq2/hbase-0.90.4-cdh3u3-patch30%2B3.jar
>>>> <--- new jar
>>>>
>>>> On Thu, Jan 31, 2013 at 6:48 AM, Jean-Marc Spaggiari <
>>>> jean-marc@spaggiari.org> wrote:
>>>>>
>>>>> Hi Brandon,
>>>>>
>>>>> I faced the same issue for "HRegionInfo was null or empty" on January
>>>>> 24th and Ted replied:
>>>>>
>>>>> "Encountered problems when prefetch META table:
>>>>>
>>>>> You can ignore the warning."
>>>>>
>>>>> So I think you should focus on the last one "not listed in META or
>>>>> deployed on any region server".
>>>>>
>>>>> Have you tried hbck to see if it can fix it?
>>>>>
>>>>> JM
>>>>>
>>>>> 2013/1/31, Brandon Peskin <bp...@adobe.com>:
>>>>>> hadoop 0.20.2-cdh3u2
>>>>>> hbase 0.90.4-cdh3u2
>>>>>>
>>>>>> On January 8th I had a network event where I lost three region
>> servers.
>>>>>>
>>>>>> When they came back I had unassigned regions/regions not being served
>>>> errors
>>>>>> which I fixed with the hbck -fix
>>>>>>
>>>>>>
>>>>>>
>>>>>> Since then, however I have been getting an increasing number of these
>>>> when I
>>>>>> have clients trying to write to specific tables:
>>>>>>
>>>>>>
>>>>>> java.io.IOException: HRegionInfo was null or empty in Meta for
>>>>>> algol_profile_training_record,
>> row=algol_profile_training_record,clientcode:49128:abce6d9f-1ee2-434a-8a82-a151b7dc183f,99999999999999
>>>>>> at
>> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:142)
>>>>>> ~[hbase-0.90.4-cdh3u2.jar:na]
>>>>>> at
>> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:95)
>>>>>> ~[hbase-0.90.4-cdh3u2.jar:na]
>>>>>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:649)
>>>>>> [hbase-0.90.4-cdh3u2.jar:na]
>>>>>>
>>>>>>
>>>>>>
>>>>>> ....to the point now where I seemingly can't even write to that
>> table.
>>>>>>
>>>>>> This also coincides with the following, seen in the the hbase master
>>>> log:
>>>>>>
>>>>>> 13/01/31 02:49:55 WARN master.CatalogJanitor: REGIONINFO_QUALIFIER is
>>>> empty
>>>>>> in
>> keyvalues={algol_tmp_client_39691_20121101053400_6,,1351783324210.b703b203d6189cad224764853cbd88f7./info:server/1351785275123/Put/vlen=32,
>> algol_tmp_clientcode_39691_20121101053400_6,,1351783324210.b703b203d6189cad224764853cbd88f7./info:serverstartcode/1351785275123/Put/vlen=8}
>>>>>> 13/01/31 02:49:55 WARN master.CatalogJanitor: REGIONINFO_QUALIFIER is
>>>> empty
>>>>>> in
>> keyvalues={algol_tmp_client_39691_20121101053400_6,\x7F\xD6\xA5Pm\xF2E\x9D\x81\xB1b\xD1'\xD3\xE5\xA4,1351783324210.507bdf342aaae45644d597c272c52e02./info:server/1351785275128/Put/vlen=32,
>> algol_tmp_clientcode_39691_20121101053400_6,\x7F\xD6\xA5Pm\xF2E\x9D\x81\xB1b\xD1'\xD3\xE5\xA4,1351783324210.507bdf342aaae45644d597c272c52e02./info:serverstartcode/1351785275128/Put/vlen=8}
>>>>>> 13/01/31 02:49:55 WARN master.CatalogJanitor: REGIONINFO_QUALIFIER is
>>>> empty
>>>>>> in
>> keyvalues={algol_tmp_client_34785_20121004102759_0,,1349383219155.a45774fa4c8af765df87d3373d30cfdc./info:server/1349461385375/Put/vlen=32,
>> algol_tmp_overstock_34785_20121004102759_0,,1349383219155.a45774fa4c8af765df87d3373d30cfdc./info:serverstartcode/1349461385375/Put/vlen=8}
>>>>>>
>>>>>>
>>>>>> ...and there's 58 of these I cannot fix (presumably I could with the
>>>> hbck
>>>>>> provided with CDH4):
>>>>>>
>>>>>> ERROR: Region
>> hdfs://namenode:9000/hbase/algol_profile_training_record/fdaa1024d1b725b4997d2283640f0fa4
>>>>>> on HDFS, but not listed in META or deployed on any region server
>>>>>>
>>>>>>
>>>>>> I'm absolutely stumped. I've done some poking around and I can't find
>>>> any
>>>>>> sort of data surrounding this issue with the exception of similar
>>>> symptoms
>>>>>> in an exchange on this list of March this year (though the inquirer
>> had
>>>>>> different questions). Any help would be appreciated, though I
>> suspect I
>>>> will
>>>>>> be told 'Upgrade to CDH4' or 'Drop and re-create the table'.
>>>>>>
>>>>>> Thanks in advance.
>>>>>>
>>>>>> --
>>>>>> Brandon Peskin
>>>>>> Senior Systems Administrator
>>>>>> Adobe Systems
>>>>>> bpeskin@adobe.com
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Kevin O'Dell
>>>> Customer Operations Engineer, Cloudera
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
Re: HRegionInfo was null or empty in Meta errors
Posted by Kevin O'dell <ke...@cloudera.com>.
Ted,
This is the second or third time I have seen this. I think it should
apply fairly clean. What do you think?
On Sat, Mar 2, 2013 at 10:47 AM, Ted Yu <yu...@gmail.com> wrote:
> HBASE-5837 is only in 0.95 and later.
>
> Do you want HBASE-5837 to be backported ?
>
> Thanks
>
> On Thu, Feb 7, 2013 at 3:17 PM, Brandon Peskin <bp...@adobe.com> wrote:
>
> > Thanks Kevin.
> >
> > Before I tried your advice, I tried this:
> >
> > scan '.META.', { FILTER =>
> > org.apache.hadoop.hbase.filter.SingleColumnValueFilter.new
> > (org.apache.hadoop.hbase.util.Bytes.toBytes('info'),
> > org.apache.hadoop.hbase.util.Bytes.toBytes('regioninfo'),
> >
> >
> org.apache.hadoop.hbase.filter.CompareFilter::CompareOp.valueOf('NOT_EQUAL'),
> > org.apache.hadoop.hbase.filter.SubstringComparator.new('algol'))}
> >
> > deleteall '.META.', '<row_key>'
> >
> >
> > The problem is at some point I fat-fingereda row key and believe I hit
> > HBASE-5837
> >
> > https://issues.apache.org/jira/browse/HBASE-5837
> >
> > I'm getting java.io.IOException: java.io.IOException:
> > java.lang.IllegalArgumentException: No 44 in
> > <2e40c841-af5b-4a5e-be0f-e06a953f05cc,1359958540596>, length=13,
> offset=37
> > Caused my master to die, can't restart it.
> >
> > Is there any way around this or have I completely hosed my hbase
> > installation?
> >
> >
> > On Jan 31, 2013, at 6:23 AM, Kevin O'dell <ke...@cloudera.com>
> > wrote:
> >
> > > I am going to disagree with ignoring the error. You will encounter
> > > failures when doing other operations such as import/exports. The first
> > > thing I would do is like JM said, lets focus on the region that is not
> in
> > > META(we at least want 0 inconsistencies). Can you please run hbck
> > -repair
> > > and then run another -details and let us know if you are still seeing
> > > errors? After that, if you are still getting the NULL errors for
> > > hregion:info in META. Can you please run echo "scan '.META.'" | hbase
> > shell
> > >> meta.out and attach the meta.out file. I would like to take a look at
> > > some of these.
> > >
> > > To be able to run the -repair we will want to use a different jar and
> > some
> > > instructions:
> > >
> > >
> > > 1. Move the new uber jar on to the system.
> > > hbase-0.90.4-cdh3u3-patch30+3.jar
> > > 2. Copy the hbase dir(/usr/lib/hbase) into /tmp/hbase dir.
> > > 3. Move the hbase jar(hbase-0.90.4-cdh3u2.jar) to a .old from the tmp
> > > and replace it with the uber-hbck(hbase-0.90.4-cdh3u3-patch30+3.jar).
> > > 4. break the sym links that directory
> > > 5. Add the value of fs.default.name from the core-site.xml to the
> > > HBase-site.xml
> > > 6. export HBASE_HOME=/tmp/hbase/ and run ./bin/hbase hbck -details
> 2>&1
> > > | tee details.out.
> > > 7. Check the details.out and make sure you are still seeing
> > > inconsistencies
> > > 8. ./bin/hbase hbck -repair 2>&1 | tee repair.out.
> > > 9. Run -details again and make sure we have 0 inconsistencies.
> > >
> > >
> > >
> >
> https://www.dropbox.com/s/fxotosglrrl1tq2/hbase-0.90.4-cdh3u3-patch30%2B3.jar
> > > <--- new jar
> > >
> > > On Thu, Jan 31, 2013 at 6:48 AM, Jean-Marc Spaggiari <
> > > jean-marc@spaggiari.org> wrote:
> > >>
> > >> Hi Brandon,
> > >>
> > >> I faced the same issue for "HRegionInfo was null or empty" on January
> > >> 24th and Ted replied:
> > >>
> > >> "Encountered problems when prefetch META table:
> > >>
> > >> You can ignore the warning."
> > >>
> > >> So I think you should focus on the last one "not listed in META or
> > >> deployed on any region server".
> > >>
> > >> Have you tried hbck to see if it can fix it?
> > >>
> > >> JM
> > >>
> > >> 2013/1/31, Brandon Peskin <bp...@adobe.com>:
> > >>> hadoop 0.20.2-cdh3u2
> > >>> hbase 0.90.4-cdh3u2
> > >>>
> > >>> On January 8th I had a network event where I lost three region
> servers.
> > >>>
> > >>> When they came back I had unassigned regions/regions not being served
> > > errors
> > >>> which I fixed with the hbck -fix
> > >>>
> > >>>
> > >>>
> > >>> Since then, however I have been getting an increasing number of these
> > > when I
> > >>> have clients trying to write to specific tables:
> > >>>
> > >>>
> > >>> java.io.IOException: HRegionInfo was null or empty in Meta for
> > >>> algol_profile_training_record,
> > >>>
> > >
> >
> row=algol_profile_training_record,clientcode:49128:abce6d9f-1ee2-434a-8a82-a151b7dc183f,99999999999999
> > >>> at
> > >>>
> > >
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:142)
> > >>> ~[hbase-0.90.4-cdh3u2.jar:na]
> > >>> at
> > >
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:95)
> > >>> ~[hbase-0.90.4-cdh3u2.jar:na]
> > >>> at
> > >>>
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:649)
> > >>> [hbase-0.90.4-cdh3u2.jar:na]
> > >>>
> > >>>
> > >>>
> > >>> ....to the point now where I seemingly can't even write to that
> table.
> > >>>
> > >>> This also coincides with the following, seen in the the hbase master
> > > log:
> > >>>
> > >>> 13/01/31 02:49:55 WARN master.CatalogJanitor: REGIONINFO_QUALIFIER is
> > > empty
> > >>> in
> > >>>
> > >
> >
> keyvalues={algol_tmp_client_39691_20121101053400_6,,1351783324210.b703b203d6189cad224764853cbd88f7./info:server/1351785275123/Put/vlen=32,
> > >>>
> > >
> >
> algol_tmp_clientcode_39691_20121101053400_6,,1351783324210.b703b203d6189cad224764853cbd88f7./info:serverstartcode/1351785275123/Put/vlen=8}
> > >>> 13/01/31 02:49:55 WARN master.CatalogJanitor: REGIONINFO_QUALIFIER is
> > > empty
> > >>> in
> > >>>
> > >
> >
> keyvalues={algol_tmp_client_39691_20121101053400_6,\x7F\xD6\xA5Pm\xF2E\x9D\x81\xB1b\xD1'\xD3\xE5\xA4,1351783324210.507bdf342aaae45644d597c272c52e02./info:server/1351785275128/Put/vlen=32,
> > >>>
> > >
> >
> algol_tmp_clientcode_39691_20121101053400_6,\x7F\xD6\xA5Pm\xF2E\x9D\x81\xB1b\xD1'\xD3\xE5\xA4,1351783324210.507bdf342aaae45644d597c272c52e02./info:serverstartcode/1351785275128/Put/vlen=8}
> > >>> 13/01/31 02:49:55 WARN master.CatalogJanitor: REGIONINFO_QUALIFIER is
> > > empty
> > >>> in
> > >>>
> > >
> >
> keyvalues={algol_tmp_client_34785_20121004102759_0,,1349383219155.a45774fa4c8af765df87d3373d30cfdc./info:server/1349461385375/Put/vlen=32,
> > >>>
> > >
> >
> algol_tmp_overstock_34785_20121004102759_0,,1349383219155.a45774fa4c8af765df87d3373d30cfdc./info:serverstartcode/1349461385375/Put/vlen=8}
> > >>>
> > >>>
> > >>> ...and there's 58 of these I cannot fix (presumably I could with the
> > > hbck
> > >>> provided with CDH4):
> > >>>
> > >>> ERROR: Region
> > >>>
> > >
> >
> hdfs://namenode:9000/hbase/algol_profile_training_record/fdaa1024d1b725b4997d2283640f0fa4
> > >>> on HDFS, but not listed in META or deployed on any region server
> > >>>
> > >>>
> > >>> I'm absolutely stumped. I've done some poking around and I can't find
> > > any
> > >>> sort of data surrounding this issue with the exception of similar
> > > symptoms
> > >>> in an exchange on this list of March this year (though the inquirer
> had
> > >>> different questions). Any help would be appreciated, though I
> suspect I
> > > will
> > >>> be told 'Upgrade to CDH4' or 'Drop and re-create the table'.
> > >>>
> > >>> Thanks in advance.
> > >>>
> > >>> --
> > >>> Brandon Peskin
> > >>> Senior Systems Administrator
> > >>> Adobe Systems
> > >>> bpeskin@adobe.com
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >
> > >
> > >
> > >
> > > --
> > > Kevin O'Dell
> > > Customer Operations Engineer, Cloudera
> >
> >
>
--
Kevin O'Dell
Customer Operations Engineer, Cloudera