You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by 茅旭峰 <m9...@gmail.com> on 2011/04/11 13:50:58 UTC
The whole key space is not covered by the .META.
Hi,
Is it possible that some table cannot cover the whole key space. What I saw
was like
====
hbase(main):006:0> put 'table1', 'abc', 'cfEStore:dasd', '123'
0 row(s) in 0.3030 seconds
hbase(main):007:0> put 'table1', 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
'cfEStore:dasd', '123'
ERROR: java.io.IOException: HRegionInfo was null or empty in .META.,
row=keyvalues={table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:server/1300672190144/Put/vlen=14,
table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:serverstartcode/1300672190144/Put/vlen=8}
Here is some help for this command:
Put a cell 'value' at specified table/row/column and optionally
timestamp coordinates. To put a cell value into table 't1' at
row 'r1' under column 'c1' marked with the time 'ts1', do:
hbase> put 't1', 'r1', 'c1', 'value', ts1
====
I guess this means our table has lost some key range for the specific
key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
is this reasonable? I wonder what might cause this issue? How can I check
it?
One weird thing, I can see such lines like
====
table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,1300519432575.0bdd3d8fa7fc710860a4ee51fc9c8625.
LC3MILeAUy8HmRFgU5-ESE-9T7w= LD4jOJWFyt4m7A3KGFST6d-uj3A=
====
from the web UI, which I think means the table has holds the key range for
key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='
I also have not seen any errors/warnings in the log file. Any suggestions?
Thanks in advance.
Re: The whole key space is not covered by the .META.
Posted by Stack <st...@duboce.net>.
2011/4/11 茅旭峰 <m9...@gmail.com>:
> We are using hadoop-CDH3B4 and hbase0.90.1-CDH3B4. I'll check the
> issue further, but my understanding is the meta info and the root
> region are saved by zookeeper, right? Do I need to check them there?
>
The .META. table is like any other and stored out on the cluster just
as user space regions are.
St.Ack
Re: The whole key space is not covered by the .META.
Posted by 茅旭峰 <m9...@gmail.com>.
The hbck output lists several empty REGIONINFO_QUALIFIER rows in .META.
I was not able to insert keys around these regions. I removed those records
in .META.
Insertion still does not work. Then we have some holes in .META.
I tried bin/hbase org.jruby.Main bin/check_meta.rb but it complained
==
11/04/13 22:57:50 INFO zookeeper.ClientCnxn: Session establishment complete
on server cloud135/10.241.67.22:2181, sessionid = 0x12edcaa37f9005f,
negotiated timeout = 40000
11/04/13 22:57:50 DEBUG client.HConnectionManager$HConnectionImplementation:
Lookedup root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@50b2fb1e;
hsa=cloud140:60020
11/04/13 22:57:50 DEBUG client.HConnectionManager$HConnectionImplementation:
Cached location for .META.,,1.1028785192 is cloud136:60020
11/04/13 22:57:50 DEBUG client.HTable$ClientScanner: Creating scanner over
.META. starting at key ''
11/04/13 22:57:50 DEBUG client.HTable$ClientScanner: Advancing internal
scanner to startKey at ''
11/04/13 22:57:50 DEBUG client.HConnectionManager$HConnectionImplementation:
Cache hit for row <> in tableName .META.: location server cloud136:60020,
location region name .META.,,1.1028785192
Writables.java:75:in `org.apache.hadoop.hbase.util.Writables.getWritable':
java.lang.NullPointerException: null (NativeException)
from Writables.java:119:in
`org.apache.hadoop.hbase.util.Writables.getHRegionInfo'
from null:-1:in `sun.reflect.GeneratedMethodAccessor6.invoke'
from DelegatingMethodAccessorImpl.java:25:in
`sun.reflect.DelegatingMethodAccessorImpl.invoke'
from Method.java:597:in `java.lang.reflect.Method.invoke'
from JavaMethod.java:196:in
`org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling'
from JavaMethod.java:182:in `org.jruby.javasupport.JavaMethod.invoke_static'
from JavaClass.java:371:in
`org.jruby.javasupport.JavaClass$StaticMethodInvoker.execute'
from SimpleCallbackMethod.java:81:in
`org.jruby.internal.runtime.methods.SimpleCallbackMethod.call'
... 16 levels...
from Main.java:183:in `org.jruby.Main.runInterpreter'
from Main.java:120:in `org.jruby.Main.run'
from Main.java:95:in `org.jruby.Main.main'
Complete Java stackTrace
java.lang.NullPointerException
at org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:75)
at org.apache.hadoop.hbase.util.Writables.getHRegionInfo(Writables.java:119)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling(JavaMethod.java:196)
at org.jruby.javasupport.JavaMethod.invoke_static(JavaMethod.java:182)
at
org.jruby.javasupport.JavaClass$StaticMethodInvoker.execute(JavaClass.java:371)
at
org.jruby.internal.runtime.methods.SimpleCallbackMethod.call(SimpleCallbackMethod.java:81)
at org.jruby.evaluator.EvaluationState.callNode(EvaluationState.java:571)
at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:207)
at
org.jruby.evaluator.EvaluationState.localAsgnNode(EvaluationState.java:1254)
at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:286)
at org.jruby.evaluator.EvaluationState.blockNode(EvaluationState.java:533)
at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:201)
at org.jruby.evaluator.EvaluationState.whileNode(EvaluationState.java:1793)
at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:387)
at org.jruby.evaluator.EvaluationState.blockNode(EvaluationState.java:533)
at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:201)
at org.jruby.evaluator.EvaluationState.rootNode(EvaluationState.java:1628)
at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:356)
at org.jruby.evaluator.EvaluationState.eval(EvaluationState.java:164)
at org.jruby.Ruby.eval(Ruby.java:278)
at org.jruby.Ruby.compileOrFallbackAndRun(Ruby.java:306)
at org.jruby.Main.runInterpreter(Main.java:238)
at org.jruby.Main.runInterpreter(Main.java:183)
at org.jruby.Main.run(Main.java:120)
at org.jruby.Main.main(Main.java:95)
==
>From a previous thread, I saw some one insert some records into .META.
manually,
to fill the holes, it seems like we need to generate encoded region name
with
tablename, start_key, id and md5-hash, I do not know how to generate id and
start_key.
Any tips on fill those region holes?
Thanks and regards,
Mao Xu-Feng
2011/4/13 茅旭峰 <m9...@gmail.com>
> Hi Stack,
>
> Sorry for bothering you in private.
>
> Regarding this issue, so far I still can not put specific key into the
> hbase?
>
> I attached the output of './bin/hbase shell' with this message.
>
> No matter what causes this issue, is there any operation I can perform to
> cover this scenario, at least we need chance to put those key-values.
>
> And then, I'm trying to find the root cause by debugging.
>
> Will keep you update. Thanks!
>
>
> 2011/4/13 Stack <st...@duboce.net>
>
>> You can also cat the .regioninfo file that is under the region
>> directory to learn more about the region -- its HRegionInfo (The file
>> format is serialized HRegionInfo followed by a String version so
>> intelligible to non-machines).
>> St.Ack
>>
>> 2011/4/12 Stack <st...@duboce.net>:
>> > These regions that are in hdfs but not in .META. and not on any region
>> > server are probably harmless. Would be interesting to trace how they
>> > got to this state. My guess is that they are old let go regions that
>> > were not cleaned up.
>> >
>> > The region directory name is its encoded name which is the tail part
>> > of the row name in the .META. table.
>> >
>> > St.Ack
>> >
>> >
>> > 2011/4/11 茅旭峰 <m9...@gmail.com>:
>> >> the results of ./bin/hbase hbck show lots of 'inconsistence status'
>> like
>> >>
>> >> ====
>> >> ERROR: Region
>> >> hdfs://cloud137:9000/hbase/table1/01c80f8b54523ad6c242c5f695544f16 on
>> HDFS,
>> >> but not listed in META or deployed on any region server.
>> >> ERROR: Region
>> >> hdfs://cloud137:9000/hbase/table1/01ce4e2f72baa0df51b7b2010c9d1ef0 on
>> HDFS,
>> >> but not listed in META or deployed on any region server.
>> >> ERROR: Region
>> >> hdfs://cloud137:9000/hbase/table1/01d0855af5e646823bdff82fddadd81e on
>> HDFS,
>> >> but not listed in META or deployed on any region server.
>> >> ====
>> >>
>> >> How can I match the names of the directories from the .META.?
>> >>
>> >> I checked the help in hbase shell, but looks like it only has
>> close_region,
>> >> no open_region.
>> >>
>> >> 2011/4/12 Stack <st...@duboce.net>
>> >>>
>> >>> Can you open the region again? (See shell commands for opening
>> regions).
>> >>>
>> >>> What does hbck say: ./bin/hbase hbck.
>> >>>
>> >>> Add the -details flag.
>> >>>
>> >>> It might tell you a story about an offlined region.
>> >>>
>> >>> 0.90.2 has some fixes for issues in and around here (CDH3 release, out
>> >>> on the 14th, has most of them bundled).
>> >>>
>> >>> St.Ack
>> >>>
>> >>> 2011/4/11 茅旭峰 <m9...@gmail.com>:
>> >>> > From the output of scan '.META.' I pasted before, we can see there
>> are
>> >>> > two
>> >>> > key ranges
>> >>> > which might cover the put key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='. They
>> are
>> >>> >
>> >>> > #1, 'LC3MILeAUy8HmRFgU5-ESE-9T7w=' -> 'LD4jOJWFyt4m7A3KGFST6d-uj3A='
>> >>> > #2, 'LC_vN8JYweYYsnKaKbpOo67kUNA=' -> 'some end key'
>> >>> >
>> >>> > The output has less info about the second key range. And I also
>> found
>> >>> > some
>> >>> > log like
>> >>> >
>> >>> > ====
>> >>> >
>> >>> >
>> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>> >>> > 09:48:36,953 DEBUG
>> >>> > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler:
>> >>> > Processing
>> >>> > close of
>> >>> >
>> >>> >
>> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
>> >>> >
>> >>> >
>> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>> >>> > 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
>> Closing
>> >>> >
>> >>> >
>> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.:
>> >>> > disabling compactions & flushes
>> >>> >
>> >>> >
>> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>> >>> > 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
>> Updates
>> >>> > disabled for region
>> >>> >
>> >>> >
>> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
>> >>> >
>> >>> >
>> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>> >>> > 09:48:36,953 INFO org.apache.hadoop.hbase.regionserver.HRegion:
>> Closed
>> >>> >
>> >>> >
>> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
>> >>> >
>> >>> >
>> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>> >>> > 09:48:36,953 DEBUG
>> >>> > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler:
>> Closed
>> >>> > region
>> >>> >
>> >>> >
>> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
>> >>> > ====
>> >>> >
>> >>> > It looks like the second had already been closed for some reason. Is
>> it
>> >>> > possile that the info of that region
>> >>> > is not clean clearly? How can I check this in detail?
>> >>> >
>> >>> > I'm guessing maybe 'getClosestRowBefore' returns the closed region
>> in
>> >>> > HConnectionManager.locateRegionInMeta,
>> >>> > which is closed, so this leads to the problem.
>> >>> >
>> >>> > 2011/4/12 茅旭峰 <m9...@gmail.com>
>> >>> >
>> >>> >> We are using hadoop-CDH3B4 and hbase0.90.1-CDH3B4. I'll check the
>> >>> >> issue further, but my understanding is the meta info and the root
>> >>> >> region are saved by zookeeper, right? Do I need to check them
>> there?
>> >>> >>
>> >>> >> m9suns
>> >>> >>
>> >>> >> 在 2011-4-12，0:40，Jean-Daniel Cryans <jd...@apache.org> 写道：
>> >>> >>
>> >>> >> > It's possible under some bugs, which HBase version are you using?
>> >>> >> >
>> >>> >> > J-D
>> >>> >> >
>> >>> >> > On Mon, Apr 11, 2011 at 4:50 AM, 茅旭峰 <m9...@gmail.com> wrote:
>> >>> >> >> Hi,
>> >>> >> >>
>> >>> >> >> Is it possible that some table cannot cover the whole key space.
>> >>> >> >> What I
>> >>> >> saw
>> >>> >> >> was like
>> >>> >> >>
>> >>> >> >> ====
>> >>> >> >> hbase(main):006:0> put 'table1', 'abc', 'cfEStore:dasd', '123'
>> >>> >> >>
>> >>> >> >> 0 row(s) in 0.3030 seconds
>> >>> >> >>
>> >>> >> >> hbase(main):007:0> put 'table1', 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
>> >>> >> >> 'cfEStore:dasd', '123'
>> >>> >> >>
>> >>> >> >> ERROR: java.io.IOException: HRegionInfo was null or empty in
>> .META.,
>> >>> >> >>
>> >>> >>
>> >>> >>
>> row=keyvalues={table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:server/1300672190144/Put/vlen=14,
>> >>> >> >>
>> >>> >>
>> >>> >>
>> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:serverstartcode/1300672190144/Put/vlen=8}
>> >>> >> >>
>> >>> >> >> Here is some help for this command:
>> >>> >> >> Put a cell 'value' at specified table/row/column and optionally
>> >>> >> >> timestamp coordinates. To put a cell value into table 't1' at
>> >>> >> >> row 'r1' under column 'c1' marked with the time 'ts1', do:
>> >>> >> >>
>> >>> >> >> hbase> put 't1', 'r1', 'c1', 'value', ts1
>> >>> >> >>
>> >>> >> >> ====
>> >>> >> >>
>> >>> >> >> I guess this means our table has lost some key range for the
>> >>> >> >> specific
>> >>> >> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
>> >>> >> >> is this reasonable? I wonder what might cause this issue? How
>> can I
>> >>> >> check
>> >>> >> >> it?
>> >>> >> >>
>> >>> >> >> One weird thing, I can see such lines like
>> >>> >> >>
>> >>> >> >> ====
>> >>> >> >>
>> >>> >>
>> >>> >>
>> table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,1300519432575.0bdd3d8fa7fc710860a4ee51fc9c8625.
>> >>> >> >> LC3MILeAUy8HmRFgU5-ESE-9T7w= LD4jOJWFyt4m7A3KGFST6d-uj3A=
>> >>> >> >> ====
>> >>> >> >>
>> >>> >> >> from the web UI, which I think means the table has holds the key
>> >>> >> >> range
>> >>> >> for
>> >>> >> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='
>> >>> >> >>
>> >>> >> >> I also have not seen any errors/warnings in the log file. Any
>> >>> >> suggestions?
>> >>> >> >>
>> >>> >> >> Thanks in advance.
>> >>> >> >>
>> >>> >>
>> >>> >
>> >>
>> >>
>> >
>>
>
>
Re: The whole key space is not covered by the .META.
Posted by Stack <st...@duboce.net>.
You can also cat the .regioninfo file that is under the region
directory to learn more about the region -- its HRegionInfo (The file
format is serialized HRegionInfo followed by a String version so
intelligible to non-machines).
St.Ack
2011/4/12 Stack <st...@duboce.net>:
> These regions that are in hdfs but not in .META. and not on any region
> server are probably harmless. Would be interesting to trace how they
> got to this state. My guess is that they are old let go regions that
> were not cleaned up.
>
> The region directory name is its encoded name which is the tail part
> of the row name in the .META. table.
>
> St.Ack
>
>
> 2011/4/11 茅旭峰 <m9...@gmail.com>:
>> the results of ./bin/hbase hbck show lots of 'inconsistence status' like
>>
>> ====
>> ERROR: Region
>> hdfs://cloud137:9000/hbase/table1/01c80f8b54523ad6c242c5f695544f16 on HDFS,
>> but not listed in META or deployed on any region server.
>> ERROR: Region
>> hdfs://cloud137:9000/hbase/table1/01ce4e2f72baa0df51b7b2010c9d1ef0 on HDFS,
>> but not listed in META or deployed on any region server.
>> ERROR: Region
>> hdfs://cloud137:9000/hbase/table1/01d0855af5e646823bdff82fddadd81e on HDFS,
>> but not listed in META or deployed on any region server.
>> ====
>>
>> How can I match the names of the directories from the .META.?
>>
>> I checked the help in hbase shell, but looks like it only has close_region,
>> no open_region.
>>
>> 2011/4/12 Stack <st...@duboce.net>
>>>
>>> Can you open the region again? (See shell commands for opening regions).
>>>
>>> What does hbck say: ./bin/hbase hbck.
>>>
>>> Add the -details flag.
>>>
>>> It might tell you a story about an offlined region.
>>>
>>> 0.90.2 has some fixes for issues in and around here (CDH3 release, out
>>> on the 14th, has most of them bundled).
>>>
>>> St.Ack
>>>
>>> 2011/4/11 茅旭峰 <m9...@gmail.com>:
>>> > From the output of scan '.META.' I pasted before, we can see there are
>>> > two
>>> > key ranges
>>> > which might cover the put key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='. They are
>>> >
>>> > #1, 'LC3MILeAUy8HmRFgU5-ESE-9T7w=' -> 'LD4jOJWFyt4m7A3KGFST6d-uj3A='
>>> > #2, 'LC_vN8JYweYYsnKaKbpOo67kUNA=' -> 'some end key'
>>> >
>>> > The output has less info about the second key range. And I also found
>>> > some
>>> > log like
>>> >
>>> > ====
>>> >
>>> > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>>> > 09:48:36,953 DEBUG
>>> > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler:
>>> > Processing
>>> > close of
>>> >
>>> > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
>>> >
>>> > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>>> > 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing
>>> >
>>> > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.:
>>> > disabling compactions & flushes
>>> >
>>> > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>>> > 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates
>>> > disabled for region
>>> >
>>> > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
>>> >
>>> > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>>> > 09:48:36,953 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed
>>> >
>>> > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
>>> >
>>> > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>>> > 09:48:36,953 DEBUG
>>> > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed
>>> > region
>>> >
>>> > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
>>> > ====
>>> >
>>> > It looks like the second had already been closed for some reason. Is it
>>> > possile that the info of that region
>>> > is not clean clearly? How can I check this in detail?
>>> >
>>> > I'm guessing maybe 'getClosestRowBefore' returns the closed region in
>>> > HConnectionManager.locateRegionInMeta,
>>> > which is closed, so this leads to the problem.
>>> >
>>> > 2011/4/12 茅旭峰 <m9...@gmail.com>
>>> >
>>> >> We are using hadoop-CDH3B4 and hbase0.90.1-CDH3B4. I'll check the
>>> >> issue further, but my understanding is the meta info and the root
>>> >> region are saved by zookeeper, right? Do I need to check them there?
>>> >>
>>> >> m9suns
>>> >>
>>> >> 在 2011-4-12，0:40，Jean-Daniel Cryans <jd...@apache.org> 写道：
>>> >>
>>> >> > It's possible under some bugs, which HBase version are you using?
>>> >> >
>>> >> > J-D
>>> >> >
>>> >> > On Mon, Apr 11, 2011 at 4:50 AM, 茅旭峰 <m9...@gmail.com> wrote:
>>> >> >> Hi,
>>> >> >>
>>> >> >> Is it possible that some table cannot cover the whole key space.
>>> >> >> What I
>>> >> saw
>>> >> >> was like
>>> >> >>
>>> >> >> ====
>>> >> >> hbase(main):006:0> put 'table1', 'abc', 'cfEStore:dasd', '123'
>>> >> >>
>>> >> >> 0 row(s) in 0.3030 seconds
>>> >> >>
>>> >> >> hbase(main):007:0> put 'table1', 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
>>> >> >> 'cfEStore:dasd', '123'
>>> >> >>
>>> >> >> ERROR: java.io.IOException: HRegionInfo was null or empty in .META.,
>>> >> >>
>>> >>
>>> >> row=keyvalues={table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:server/1300672190144/Put/vlen=14,
>>> >> >>
>>> >>
>>> >> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:serverstartcode/1300672190144/Put/vlen=8}
>>> >> >>
>>> >> >> Here is some help for this command:
>>> >> >> Put a cell 'value' at specified table/row/column and optionally
>>> >> >> timestamp coordinates. To put a cell value into table 't1' at
>>> >> >> row 'r1' under column 'c1' marked with the time 'ts1', do:
>>> >> >>
>>> >> >> hbase> put 't1', 'r1', 'c1', 'value', ts1
>>> >> >>
>>> >> >> ====
>>> >> >>
>>> >> >> I guess this means our table has lost some key range for the
>>> >> >> specific
>>> >> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
>>> >> >> is this reasonable? I wonder what might cause this issue? How can I
>>> >> check
>>> >> >> it?
>>> >> >>
>>> >> >> One weird thing, I can see such lines like
>>> >> >>
>>> >> >> ====
>>> >> >>
>>> >>
>>> >> table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,1300519432575.0bdd3d8fa7fc710860a4ee51fc9c8625.
>>> >> >> LC3MILeAUy8HmRFgU5-ESE-9T7w= LD4jOJWFyt4m7A3KGFST6d-uj3A=
>>> >> >> ====
>>> >> >>
>>> >> >> from the web UI, which I think means the table has holds the key
>>> >> >> range
>>> >> for
>>> >> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='
>>> >> >>
>>> >> >> I also have not seen any errors/warnings in the log file. Any
>>> >> suggestions?
>>> >> >>
>>> >> >> Thanks in advance.
>>> >> >>
>>> >>
>>> >
>>
>>
>
Re: The whole key space is not covered by the .META.
Posted by Stack <st...@duboce.net>.
These regions that are in hdfs but not in .META. and not on any region
server are probably harmless. Would be interesting to trace how they
got to this state. My guess is that they are old let go regions that
were not cleaned up.
The region directory name is its encoded name which is the tail part
of the row name in the .META. table.
St.Ack
2011/4/11 茅旭峰 <m9...@gmail.com>:
> the results of ./bin/hbase hbck show lots of 'inconsistence status' like
>
> ====
> ERROR: Region
> hdfs://cloud137:9000/hbase/table1/01c80f8b54523ad6c242c5f695544f16 on HDFS,
> but not listed in META or deployed on any region server.
> ERROR: Region
> hdfs://cloud137:9000/hbase/table1/01ce4e2f72baa0df51b7b2010c9d1ef0 on HDFS,
> but not listed in META or deployed on any region server.
> ERROR: Region
> hdfs://cloud137:9000/hbase/table1/01d0855af5e646823bdff82fddadd81e on HDFS,
> but not listed in META or deployed on any region server.
> ====
>
> How can I match the names of the directories from the .META.?
>
> I checked the help in hbase shell, but looks like it only has close_region,
> no open_region.
>
> 2011/4/12 Stack <st...@duboce.net>
>>
>> Can you open the region again? (See shell commands for opening regions).
>>
>> What does hbck say: ./bin/hbase hbck.
>>
>> Add the -details flag.
>>
>> It might tell you a story about an offlined region.
>>
>> 0.90.2 has some fixes for issues in and around here (CDH3 release, out
>> on the 14th, has most of them bundled).
>>
>> St.Ack
>>
>> 2011/4/11 茅旭峰 <m9...@gmail.com>:
>> > From the output of scan '.META.' I pasted before, we can see there are
>> > two
>> > key ranges
>> > which might cover the put key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='. They are
>> >
>> > #1, 'LC3MILeAUy8HmRFgU5-ESE-9T7w=' -> 'LD4jOJWFyt4m7A3KGFST6d-uj3A='
>> > #2, 'LC_vN8JYweYYsnKaKbpOo67kUNA=' -> 'some end key'
>> >
>> > The output has less info about the second key range. And I also found
>> > some
>> > log like
>> >
>> > ====
>> >
>> > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>> > 09:48:36,953 DEBUG
>> > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler:
>> > Processing
>> > close of
>> >
>> > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
>> >
>> > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>> > 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing
>> >
>> > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.:
>> > disabling compactions & flushes
>> >
>> > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>> > 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates
>> > disabled for region
>> >
>> > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
>> >
>> > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>> > 09:48:36,953 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed
>> >
>> > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
>> >
>> > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>> > 09:48:36,953 DEBUG
>> > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed
>> > region
>> >
>> > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
>> > ====
>> >
>> > It looks like the second had already been closed for some reason. Is it
>> > possile that the info of that region
>> > is not clean clearly? How can I check this in detail?
>> >
>> > I'm guessing maybe 'getClosestRowBefore' returns the closed region in
>> > HConnectionManager.locateRegionInMeta,
>> > which is closed, so this leads to the problem.
>> >
>> > 2011/4/12 茅旭峰 <m9...@gmail.com>
>> >
>> >> We are using hadoop-CDH3B4 and hbase0.90.1-CDH3B4. I'll check the
>> >> issue further, but my understanding is the meta info and the root
>> >> region are saved by zookeeper, right? Do I need to check them there?
>> >>
>> >> m9suns
>> >>
>> >> 在 2011-4-12，0:40，Jean-Daniel Cryans <jd...@apache.org> 写道：
>> >>
>> >> > It's possible under some bugs, which HBase version are you using?
>> >> >
>> >> > J-D
>> >> >
>> >> > On Mon, Apr 11, 2011 at 4:50 AM, 茅旭峰 <m9...@gmail.com> wrote:
>> >> >> Hi,
>> >> >>
>> >> >> Is it possible that some table cannot cover the whole key space.
>> >> >> What I
>> >> saw
>> >> >> was like
>> >> >>
>> >> >> ====
>> >> >> hbase(main):006:0> put 'table1', 'abc', 'cfEStore:dasd', '123'
>> >> >>
>> >> >> 0 row(s) in 0.3030 seconds
>> >> >>
>> >> >> hbase(main):007:0> put 'table1', 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
>> >> >> 'cfEStore:dasd', '123'
>> >> >>
>> >> >> ERROR: java.io.IOException: HRegionInfo was null or empty in .META.,
>> >> >>
>> >>
>> >> row=keyvalues={table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:server/1300672190144/Put/vlen=14,
>> >> >>
>> >>
>> >> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:serverstartcode/1300672190144/Put/vlen=8}
>> >> >>
>> >> >> Here is some help for this command:
>> >> >> Put a cell 'value' at specified table/row/column and optionally
>> >> >> timestamp coordinates. To put a cell value into table 't1' at
>> >> >> row 'r1' under column 'c1' marked with the time 'ts1', do:
>> >> >>
>> >> >> hbase> put 't1', 'r1', 'c1', 'value', ts1
>> >> >>
>> >> >> ====
>> >> >>
>> >> >> I guess this means our table has lost some key range for the
>> >> >> specific
>> >> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
>> >> >> is this reasonable? I wonder what might cause this issue? How can I
>> >> check
>> >> >> it?
>> >> >>
>> >> >> One weird thing, I can see such lines like
>> >> >>
>> >> >> ====
>> >> >>
>> >>
>> >> table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,1300519432575.0bdd3d8fa7fc710860a4ee51fc9c8625.
>> >> >> LC3MILeAUy8HmRFgU5-ESE-9T7w= LD4jOJWFyt4m7A3KGFST6d-uj3A=
>> >> >> ====
>> >> >>
>> >> >> from the web UI, which I think means the table has holds the key
>> >> >> range
>> >> for
>> >> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='
>> >> >>
>> >> >> I also have not seen any errors/warnings in the log file. Any
>> >> suggestions?
>> >> >>
>> >> >> Thanks in advance.
>> >> >>
>> >>
>> >
>
>
Re: The whole key space is not covered by the .META.
Posted by 茅旭峰 <m9...@gmail.com>.
the results of ./bin/hbase hbck show lots of 'inconsistence status' like
====
ERROR: Region
hdfs://cloud137:9000/hbase/table1/01c80f8b54523ad6c242c5f695544f16 on HDFS,
but not listed in META or deployed on any region server.
ERROR: Region
hdfs://cloud137:9000/hbase/table1/01ce4e2f72baa0df51b7b2010c9d1ef0 on HDFS,
but not listed in META or deployed on any region server.
ERROR: Region
hdfs://cloud137:9000/hbase/table1/01d0855af5e646823bdff82fddadd81e on HDFS,
but not listed in META or deployed on any region server.
====
How can I match the names of the directories from the .META.?
I checked the help in hbase shell, but looks like it only has close_region,
no open_region.
2011/4/12 Stack <st...@duboce.net>
> Can you open the region again? (See shell commands for opening regions).
>
> What does hbck say: ./bin/hbase hbck.
>
> Add the -details flag.
>
> It might tell you a story about an offlined region.
>
> 0.90.2 has some fixes for issues in and around here (CDH3 release, out
> on the 14th, has most of them bundled).
>
> St.Ack
>
> 2011/4/11 茅旭峰 <m9...@gmail.com>:
> > From the output of scan '.META.' I pasted before, we can see there are
> two
> > key ranges
> > which might cover the put key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='. They are
> >
> > #1, 'LC3MILeAUy8HmRFgU5-ESE-9T7w=' -> 'LD4jOJWFyt4m7A3KGFST6d-uj3A='
> > #2, 'LC_vN8JYweYYsnKaKbpOo67kUNA=' -> 'some end key'
> >
> > The output has less info about the second key range. And I also found
> some
> > log like
> >
> > ====
> >
> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
> > 09:48:36,953 DEBUG
> > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler:
> Processing
> > close of
> >
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
> >
> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
> > 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing
> >
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.:
> > disabling compactions & flushes
> >
> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
> > 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates
> > disabled for region
> >
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
> >
> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
> > 09:48:36,953 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed
> >
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
> >
> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
> > 09:48:36,953 DEBUG
> > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed
> > region
> >
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
> > ====
> >
> > It looks like the second had already been closed for some reason. Is it
> > possile that the info of that region
> > is not clean clearly? How can I check this in detail?
> >
> > I'm guessing maybe 'getClosestRowBefore' returns the closed region in
> > HConnectionManager.locateRegionInMeta,
> > which is closed, so this leads to the problem.
> >
> > 2011/4/12 茅旭峰 <m9...@gmail.com>
> >
> >> We are using hadoop-CDH3B4 and hbase0.90.1-CDH3B4. I'll check the
> >> issue further, but my understanding is the meta info and the root
> >> region are saved by zookeeper, right? Do I need to check them there?
> >>
> >> m9suns
> >>
> >> 在 2011-4-12，0:40，Jean-Daniel Cryans <jd...@apache.org> 写道：
> >>
> >> > It's possible under some bugs, which HBase version are you using?
> >> >
> >> > J-D
> >> >
> >> > On Mon, Apr 11, 2011 at 4:50 AM, 茅旭峰 <m9...@gmail.com> wrote:
> >> >> Hi,
> >> >>
> >> >> Is it possible that some table cannot cover the whole key space. What
> I
> >> saw
> >> >> was like
> >> >>
> >> >> ====
> >> >> hbase(main):006:0> put 'table1', 'abc', 'cfEStore:dasd', '123'
> >> >>
> >> >> 0 row(s) in 0.3030 seconds
> >> >>
> >> >> hbase(main):007:0> put 'table1', 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
> >> >> 'cfEStore:dasd', '123'
> >> >>
> >> >> ERROR: java.io.IOException: HRegionInfo was null or empty in .META.,
> >> >>
> >>
> row=keyvalues={table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:server/1300672190144/Put/vlen=14,
> >> >>
> >>
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:serverstartcode/1300672190144/Put/vlen=8}
> >> >>
> >> >> Here is some help for this command:
> >> >> Put a cell 'value' at specified table/row/column and optionally
> >> >> timestamp coordinates. To put a cell value into table 't1' at
> >> >> row 'r1' under column 'c1' marked with the time 'ts1', do:
> >> >>
> >> >> hbase> put 't1', 'r1', 'c1', 'value', ts1
> >> >>
> >> >> ====
> >> >>
> >> >> I guess this means our table has lost some key range for the specific
> >> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
> >> >> is this reasonable? I wonder what might cause this issue? How can I
> >> check
> >> >> it?
> >> >>
> >> >> One weird thing, I can see such lines like
> >> >>
> >> >> ====
> >> >>
> >>
> table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,1300519432575.0bdd3d8fa7fc710860a4ee51fc9c8625.
> >> >> LC3MILeAUy8HmRFgU5-ESE-9T7w= LD4jOJWFyt4m7A3KGFST6d-uj3A=
> >> >> ====
> >> >>
> >> >> from the web UI, which I think means the table has holds the key
> range
> >> for
> >> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='
> >> >>
> >> >> I also have not seen any errors/warnings in the log file. Any
> >> suggestions?
> >> >>
> >> >> Thanks in advance.
> >> >>
> >>
> >
>
Re: The whole key space is not covered by the .META.
Posted by Stack <st...@duboce.net>.
Can you open the region again? (See shell commands for opening regions).
What does hbck say: ./bin/hbase hbck.
Add the -details flag.
It might tell you a story about an offlined region.
0.90.2 has some fixes for issues in and around here (CDH3 release, out
on the 14th, has most of them bundled).
St.Ack
2011/4/11 茅旭峰 <m9...@gmail.com>:
> From the output of scan '.META.' I pasted before, we can see there are two
> key ranges
> which might cover the put key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='. They are
>
> #1, 'LC3MILeAUy8HmRFgU5-ESE-9T7w=' -> 'LD4jOJWFyt4m7A3KGFST6d-uj3A='
> #2, 'LC_vN8JYweYYsnKaKbpOo67kUNA=' -> 'some end key'
>
> The output has less info about the second key range. And I also found some
> log like
>
> ====
> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
> 09:48:36,953 DEBUG
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing
> close of
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
> 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.:
> disabling compactions & flushes
> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
> 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates
> disabled for region
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
> 09:48:36,953 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
> 09:48:36,953 DEBUG
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed
> region
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
> ====
>
> It looks like the second had already been closed for some reason. Is it
> possile that the info of that region
> is not clean clearly? How can I check this in detail?
>
> I'm guessing maybe 'getClosestRowBefore' returns the closed region in
> HConnectionManager.locateRegionInMeta,
> which is closed, so this leads to the problem.
>
> 2011/4/12 茅旭峰 <m9...@gmail.com>
>
>> We are using hadoop-CDH3B4 and hbase0.90.1-CDH3B4. I'll check the
>> issue further, but my understanding is the meta info and the root
>> region are saved by zookeeper, right? Do I need to check them there?
>>
>> m9suns
>>
>> 在 2011-4-12，0:40，Jean-Daniel Cryans <jd...@apache.org> 写道：
>>
>> > It's possible under some bugs, which HBase version are you using?
>> >
>> > J-D
>> >
>> > On Mon, Apr 11, 2011 at 4:50 AM, 茅旭峰 <m9...@gmail.com> wrote:
>> >> Hi,
>> >>
>> >> Is it possible that some table cannot cover the whole key space. What I
>> saw
>> >> was like
>> >>
>> >> ====
>> >> hbase(main):006:0> put 'table1', 'abc', 'cfEStore:dasd', '123'
>> >>
>> >> 0 row(s) in 0.3030 seconds
>> >>
>> >> hbase(main):007:0> put 'table1', 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
>> >> 'cfEStore:dasd', '123'
>> >>
>> >> ERROR: java.io.IOException: HRegionInfo was null or empty in .META.,
>> >>
>> row=keyvalues={table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:server/1300672190144/Put/vlen=14,
>> >>
>> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:serverstartcode/1300672190144/Put/vlen=8}
>> >>
>> >> Here is some help for this command:
>> >> Put a cell 'value' at specified table/row/column and optionally
>> >> timestamp coordinates. To put a cell value into table 't1' at
>> >> row 'r1' under column 'c1' marked with the time 'ts1', do:
>> >>
>> >> hbase> put 't1', 'r1', 'c1', 'value', ts1
>> >>
>> >> ====
>> >>
>> >> I guess this means our table has lost some key range for the specific
>> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
>> >> is this reasonable? I wonder what might cause this issue? How can I
>> check
>> >> it?
>> >>
>> >> One weird thing, I can see such lines like
>> >>
>> >> ====
>> >>
>> table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,1300519432575.0bdd3d8fa7fc710860a4ee51fc9c8625.
>> >> LC3MILeAUy8HmRFgU5-ESE-9T7w= LD4jOJWFyt4m7A3KGFST6d-uj3A=
>> >> ====
>> >>
>> >> from the web UI, which I think means the table has holds the key range
>> for
>> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='
>> >>
>> >> I also have not seen any errors/warnings in the log file. Any
>> suggestions?
>> >>
>> >> Thanks in advance.
>> >>
>>
>
Re: The whole key space is not covered by the .META.
Posted by 茅旭峰 <m9...@gmail.com>.
My guess was the second region is closed, but its directory named of the
encoded name is still
on the hdfs, this leads to the inconsistency.
I wonder what happened when the region is closed, it looks like the process
is over, from the log.
2011/4/12 茅旭峰 <m9...@gmail.com>
> Anyway, looks like the two regions below have some kind of overlap, does
> this make sense?
>
>
> 2011/4/12 茅旭峰 <m9...@gmail.com>
>
>> From the output of scan '.META.' I pasted before, we can see there are two
>> key ranges
>> which might cover the put key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='. They are
>>
>> #1, 'LC3MILeAUy8HmRFgU5-ESE-9T7w=' -> 'LD4jOJWFyt4m7A3KGFST6d-uj3A='
>> #2, 'LC_vN8JYweYYsnKaKbpOo67kUNA=' -> 'some end key'
>>
>> The output has less info about the second key range. And I also found some
>> log like
>>
>> ====
>> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>> 09:48:36,953 DEBUG
>> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing
>> close of
>> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
>> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>> 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing
>> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.:
>> disabling compactions & flushes
>> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>> 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates
>> disabled for region
>> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
>> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>> 09:48:36,953 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed
>> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
>> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>> 09:48:36,953 DEBUG
>> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed
>> region
>> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
>> ====
>>
>> It looks like the second had already been closed for some reason. Is it
>> possile that the info of that region
>> is not clean clearly? How can I check this in detail?
>>
>> I'm guessing maybe 'getClosestRowBefore' returns the closed region in
>> HConnectionManager.locateRegionInMeta,
>> which is closed, so this leads to the problem.
>>
>>
>> 2011/4/12 茅旭峰 <m9...@gmail.com>
>>
>>> We are using hadoop-CDH3B4 and hbase0.90.1-CDH3B4. I'll check the
>>> issue further, but my understanding is the meta info and the root
>>> region are saved by zookeeper, right? Do I need to check them there?
>>>
>>> m9suns
>>>
>>> 在 2011-4-12，0:40，Jean-Daniel Cryans <jd...@apache.org> 写道：
>>>
>>> > It's possible under some bugs, which HBase version are you using?
>>> >
>>> > J-D
>>> >
>>> > On Mon, Apr 11, 2011 at 4:50 AM, 茅旭峰 <m9...@gmail.com> wrote:
>>> >> Hi,
>>> >>
>>> >> Is it possible that some table cannot cover the whole key space. What
>>> I saw
>>> >> was like
>>> >>
>>> >> ====
>>> >> hbase(main):006:0> put 'table1', 'abc', 'cfEStore:dasd', '123'
>>> >>
>>> >> 0 row(s) in 0.3030 seconds
>>> >>
>>> >> hbase(main):007:0> put 'table1', 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
>>> >> 'cfEStore:dasd', '123'
>>> >>
>>> >> ERROR: java.io.IOException: HRegionInfo was null or empty in .META.,
>>> >>
>>> row=keyvalues={table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:server/1300672190144/Put/vlen=14,
>>> >>
>>> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:serverstartcode/1300672190144/Put/vlen=8}
>>> >>
>>> >> Here is some help for this command:
>>> >> Put a cell 'value' at specified table/row/column and optionally
>>> >> timestamp coordinates. To put a cell value into table 't1' at
>>> >> row 'r1' under column 'c1' marked with the time 'ts1', do:
>>> >>
>>> >> hbase> put 't1', 'r1', 'c1', 'value', ts1
>>> >>
>>> >> ====
>>> >>
>>> >> I guess this means our table has lost some key range for the specific
>>> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
>>> >> is this reasonable? I wonder what might cause this issue? How can I
>>> check
>>> >> it?
>>> >>
>>> >> One weird thing, I can see such lines like
>>> >>
>>> >> ====
>>> >>
>>> table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,1300519432575.0bdd3d8fa7fc710860a4ee51fc9c8625.
>>> >> LC3MILeAUy8HmRFgU5-ESE-9T7w= LD4jOJWFyt4m7A3KGFST6d-uj3A=
>>> >> ====
>>> >>
>>> >> from the web UI, which I think means the table has holds the key range
>>> for
>>> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='
>>> >>
>>> >> I also have not seen any errors/warnings in the log file. Any
>>> suggestions?
>>> >>
>>> >> Thanks in advance.
>>> >>
>>>
>>
>>
>
Re: The whole key space is not covered by the .META.
Posted by 茅旭峰 <m9...@gmail.com>.
Anyway, looks like the two regions below have some kind of overlap, does
this make sense?
2011/4/12 茅旭峰 <m9...@gmail.com>
> From the output of scan '.META.' I pasted before, we can see there are two
> key ranges
> which might cover the put key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='. They are
>
> #1, 'LC3MILeAUy8HmRFgU5-ESE-9T7w=' -> 'LD4jOJWFyt4m7A3KGFST6d-uj3A='
> #2, 'LC_vN8JYweYYsnKaKbpOo67kUNA=' -> 'some end key'
>
> The output has less info about the second key range. And I also found some
> log like
>
> ====
> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
> 09:48:36,953 DEBUG
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing
> close of
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
> 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.:
> disabling compactions & flushes
> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
> 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates
> disabled for region
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
> 09:48:36,953 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
> 09:48:36,953 DEBUG
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed
> region
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
> ====
>
> It looks like the second had already been closed for some reason. Is it
> possile that the info of that region
> is not clean clearly? How can I check this in detail?
>
> I'm guessing maybe 'getClosestRowBefore' returns the closed region in
> HConnectionManager.locateRegionInMeta,
> which is closed, so this leads to the problem.
>
>
> 2011/4/12 茅旭峰 <m9...@gmail.com>
>
>> We are using hadoop-CDH3B4 and hbase0.90.1-CDH3B4. I'll check the
>> issue further, but my understanding is the meta info and the root
>> region are saved by zookeeper, right? Do I need to check them there?
>>
>> m9suns
>>
>> 在 2011-4-12，0:40，Jean-Daniel Cryans <jd...@apache.org> 写道：
>>
>> > It's possible under some bugs, which HBase version are you using?
>> >
>> > J-D
>> >
>> > On Mon, Apr 11, 2011 at 4:50 AM, 茅旭峰 <m9...@gmail.com> wrote:
>> >> Hi,
>> >>
>> >> Is it possible that some table cannot cover the whole key space. What I
>> saw
>> >> was like
>> >>
>> >> ====
>> >> hbase(main):006:0> put 'table1', 'abc', 'cfEStore:dasd', '123'
>> >>
>> >> 0 row(s) in 0.3030 seconds
>> >>
>> >> hbase(main):007:0> put 'table1', 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
>> >> 'cfEStore:dasd', '123'
>> >>
>> >> ERROR: java.io.IOException: HRegionInfo was null or empty in .META.,
>> >>
>> row=keyvalues={table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:server/1300672190144/Put/vlen=14,
>> >>
>> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:serverstartcode/1300672190144/Put/vlen=8}
>> >>
>> >> Here is some help for this command:
>> >> Put a cell 'value' at specified table/row/column and optionally
>> >> timestamp coordinates. To put a cell value into table 't1' at
>> >> row 'r1' under column 'c1' marked with the time 'ts1', do:
>> >>
>> >> hbase> put 't1', 'r1', 'c1', 'value', ts1
>> >>
>> >> ====
>> >>
>> >> I guess this means our table has lost some key range for the specific
>> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
>> >> is this reasonable? I wonder what might cause this issue? How can I
>> check
>> >> it?
>> >>
>> >> One weird thing, I can see such lines like
>> >>
>> >> ====
>> >>
>> table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,1300519432575.0bdd3d8fa7fc710860a4ee51fc9c8625.
>> >> LC3MILeAUy8HmRFgU5-ESE-9T7w= LD4jOJWFyt4m7A3KGFST6d-uj3A=
>> >> ====
>> >>
>> >> from the web UI, which I think means the table has holds the key range
>> for
>> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='
>> >>
>> >> I also have not seen any errors/warnings in the log file. Any
>> suggestions?
>> >>
>> >> Thanks in advance.
>> >>
>>
>
>
Re: The whole key space is not covered by the .META.
Posted by 茅旭峰 <m9...@gmail.com>.
>From the output of scan '.META.' I pasted before, we can see there are two
key ranges
which might cover the put key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='. They are
#1, 'LC3MILeAUy8HmRFgU5-ESE-9T7w=' -> 'LD4jOJWFyt4m7A3KGFST6d-uj3A='
#2, 'LC_vN8JYweYYsnKaKbpOo67kUNA=' -> 'some end key'
The output has less info about the second key range. And I also found some
log like
====
essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
09:48:36,953 DEBUG
org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing
close of
table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing
table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.:
disabling compactions & flushes
essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates
disabled for region
table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
09:48:36,953 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed
table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
09:48:36,953 DEBUG
org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed
region
table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
====
It looks like the second had already been closed for some reason. Is it
possile that the info of that region
is not clean clearly? How can I check this in detail?
I'm guessing maybe 'getClosestRowBefore' returns the closed region in
HConnectionManager.locateRegionInMeta,
which is closed, so this leads to the problem.
2011/4/12 茅旭峰 <m9...@gmail.com>
> We are using hadoop-CDH3B4 and hbase0.90.1-CDH3B4. I'll check the
> issue further, but my understanding is the meta info and the root
> region are saved by zookeeper, right? Do I need to check them there?
>
> m9suns
>
> 在 2011-4-12，0:40，Jean-Daniel Cryans <jd...@apache.org> 写道：
>
> > It's possible under some bugs, which HBase version are you using?
> >
> > J-D
> >
> > On Mon, Apr 11, 2011 at 4:50 AM, 茅旭峰 <m9...@gmail.com> wrote:
> >> Hi,
> >>
> >> Is it possible that some table cannot cover the whole key space. What I
> saw
> >> was like
> >>
> >> ====
> >> hbase(main):006:0> put 'table1', 'abc', 'cfEStore:dasd', '123'
> >>
> >> 0 row(s) in 0.3030 seconds
> >>
> >> hbase(main):007:0> put 'table1', 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
> >> 'cfEStore:dasd', '123'
> >>
> >> ERROR: java.io.IOException: HRegionInfo was null or empty in .META.,
> >>
> row=keyvalues={table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:server/1300672190144/Put/vlen=14,
> >>
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:serverstartcode/1300672190144/Put/vlen=8}
> >>
> >> Here is some help for this command:
> >> Put a cell 'value' at specified table/row/column and optionally
> >> timestamp coordinates. To put a cell value into table 't1' at
> >> row 'r1' under column 'c1' marked with the time 'ts1', do:
> >>
> >> hbase> put 't1', 'r1', 'c1', 'value', ts1
> >>
> >> ====
> >>
> >> I guess this means our table has lost some key range for the specific
> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
> >> is this reasonable? I wonder what might cause this issue? How can I
> check
> >> it?
> >>
> >> One weird thing, I can see such lines like
> >>
> >> ====
> >>
> table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,1300519432575.0bdd3d8fa7fc710860a4ee51fc9c8625.
> >> LC3MILeAUy8HmRFgU5-ESE-9T7w= LD4jOJWFyt4m7A3KGFST6d-uj3A=
> >> ====
> >>
> >> from the web UI, which I think means the table has holds the key range
> for
> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='
> >>
> >> I also have not seen any errors/warnings in the log file. Any
> suggestions?
> >>
> >> Thanks in advance.
> >>
>
Re: The whole key space is not covered by the .META.
Posted by 茅旭峰 <m9...@gmail.com>.
We are using hadoop-CDH3B4 and hbase0.90.1-CDH3B4. I'll check the
issue further, but my understanding is the meta info and the root
region are saved by zookeeper, right? Do I need to check them there?
m9suns
在 2011-4-12，0:40，Jean-Daniel Cryans <jd...@apache.org> 写道：
> It's possible under some bugs, which HBase version are you using?
>
> J-D
>
> On Mon, Apr 11, 2011 at 4:50 AM, 茅旭峰 <m9...@gmail.com> wrote:
>> Hi,
>>
>> Is it possible that some table cannot cover the whole key space. What I saw
>> was like
>>
>> ====
>> hbase(main):006:0> put 'table1', 'abc', 'cfEStore:dasd', '123'
>>
>> 0 row(s) in 0.3030 seconds
>>
>> hbase(main):007:0> put 'table1', 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
>> 'cfEStore:dasd', '123'
>>
>> ERROR: java.io.IOException: HRegionInfo was null or empty in .META.,
>> row=keyvalues={table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:server/1300672190144/Put/vlen=14,
>> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:serverstartcode/1300672190144/Put/vlen=8}
>>
>> Here is some help for this command:
>> Put a cell 'value' at specified table/row/column and optionally
>> timestamp coordinates. To put a cell value into table 't1' at
>> row 'r1' under column 'c1' marked with the time 'ts1', do:
>>
>> hbase> put 't1', 'r1', 'c1', 'value', ts1
>>
>> ====
>>
>> I guess this means our table has lost some key range for the specific
>> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
>> is this reasonable? I wonder what might cause this issue? How can I check
>> it?
>>
>> One weird thing, I can see such lines like
>>
>> ====
>> table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,1300519432575.0bdd3d8fa7fc710860a4ee51fc9c8625.
>> LC3MILeAUy8HmRFgU5-ESE-9T7w= LD4jOJWFyt4m7A3KGFST6d-uj3A=
>> ====
>>
>> from the web UI, which I think means the table has holds the key range for
>> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='
>>
>> I also have not seen any errors/warnings in the log file. Any suggestions?
>>
>> Thanks in advance.
>>
Re: The whole key space is not covered by the .META.
Posted by Jean-Daniel Cryans <jd...@apache.org>.
It's possible under some bugs, which HBase version are you using?
J-D
On Mon, Apr 11, 2011 at 4:50 AM, 茅旭峰 <m9...@gmail.com> wrote:
> Hi,
>
> Is it possible that some table cannot cover the whole key space. What I saw
> was like
>
> ====
> hbase(main):006:0> put 'table1', 'abc', 'cfEStore:dasd', '123'
>
> 0 row(s) in 0.3030 seconds
>
> hbase(main):007:0> put 'table1', 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
> 'cfEStore:dasd', '123'
>
> ERROR: java.io.IOException: HRegionInfo was null or empty in .META.,
> row=keyvalues={table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:server/1300672190144/Put/vlen=14,
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:serverstartcode/1300672190144/Put/vlen=8}
>
> Here is some help for this command:
> Put a cell 'value' at specified table/row/column and optionally
> timestamp coordinates. To put a cell value into table 't1' at
> row 'r1' under column 'c1' marked with the time 'ts1', do:
>
> hbase> put 't1', 'r1', 'c1', 'value', ts1
>
> ====
>
> I guess this means our table has lost some key range for the specific
> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
> is this reasonable? I wonder what might cause this issue? How can I check
> it?
>
> One weird thing, I can see such lines like
>
> ====
> table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,1300519432575.0bdd3d8fa7fc710860a4ee51fc9c8625.
> LC3MILeAUy8HmRFgU5-ESE-9T7w= LD4jOJWFyt4m7A3KGFST6d-uj3A=
> ====
>
> from the web UI, which I think means the table has holds the key range for
> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='
>
> I also have not seen any errors/warnings in the log file. Any suggestions?
>
> Thanks in advance.
>
Re: The whole key space is not covered by the .META.
Posted by 茅旭峰 <m9...@gmail.com>.
more input
from the hbase shell, I used scan '.META', I got
===
table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,130 column=info:regioninfo,
timestamp=1300519437064, value=REGION => {NAME =>
'table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,13005
0519432575.0bdd3d8fa7fc710860a4ee51fc9c
19432575.0bdd3d8fa7fc710860a4ee51fc9c8625.', STARTKEY =>
'LC3MILeAUy8HmRFgU5-ESE-9T7w=', ENDKEY => 'LD4jOJWFyt4m7A3K
8625. GFST6d-uj3A=', ENCODED =>
0bdd3d8fa7fc710860a4ee51fc9c8625, TABLE => {{NAME => 'table1', FAMILIES =>
[{NAME => 'cfES
tore', BLOOMFILTER => 'NONE',
REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'NONE', TTL =>
'2147483647',
BLOCKSIZE => '65536', IN_MEMORY =>
'false', BLOCKCACHE => 'true'}]}}
table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,130 column=info:server,
timestamp=1302521420981, value=cloud134:60020
0519432575.0bdd3d8fa7fc710860a4ee51fc9c
8625.
table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,130 column=info:serverstartcode,
timestamp=1302521420981, value=1302520792395
0519432575.0bdd3d8fa7fc710860a4ee51fc9c
8625.
table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,130 column=info:server,
timestamp=1300672190023, value=cloud140:60020
0594695996.d534c265dca68bb262e1dbe223d5
e93d.
table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,130 column=info:serverstartcode,
timestamp=1300672190023, value=1300671116560
0594695996.d534c265dca68bb262e1dbe223d5
e93d.
table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,130 column=info:server,
timestamp=1300672190144, value=cloud140:60020
0594695996.ba8e0bffb79bda039f6800f78ad2
2dcd.
table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,130 column=info:serverstartcode,
timestamp=1300672190144, value=1300671116560
0594695996.ba8e0bffb79bda039f6800f78ad2
2dcd.
table1,LD4jOJWFyt4m7A3KGFST6d-uj3A=,130 column=info:regioninfo,
timestamp=1300583209179, value=REGION => {NAME =>
'table1,LD4jOJWFyt4m7A3KGFST6d-uj3A=,13005
0583204256.b7d6e116d91d516a0c1709fb29e7
83204256.b7d6e116d91d516a0c1709fb29e765b4.', STARTKEY =>
'LD4jOJWFyt4m7A3KGFST6d-uj3A=', ENDKEY => 'LDaGUgQLscCTC477
65b4. 1l9Rq4MHORs=', ENCODED =>
b7d6e116d91d516a0c1709fb29e765b4, TABLE => {{NAME => 'table1', FAMILIES =>
[{NAME => 'cfES
tore', BLOOMFILTER => 'NONE',
REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'NONE', TTL =>
'2147483647',
BLOCKSIZE => '65536', IN_MEMORY =>
'false', BLOCKCACHE => 'true'}]}}
table1,LD4jOJWFyt4m7A3KGFST6d-uj3A=,130 column=info:server,
timestamp=1302521420744, value=cloud134:60020
0583204256.b7d6e116d91d516a0c1709fb29e7
65b4.
table1,LD4jOJWFyt4m7A3KGFST6d-uj3A=,130 column=info:serverstartcode,
timestamp=1302521420744, value=1302520792395
0583204256.b7d6e116d91d516a0c1709fb29e7
65b4.
===
Looks like we do have the region ''LC3MILeAUy8HmRFgU5-ESE-9T7w=' to
'LD4jOJWFyt4m7A3KGFST6d-uj3A='.
I've tried flush '.META.' and major compact '.META.' in the shell, as link
http://www.mail-archive.com/hbase-dev@hadoop.apache.org/msg12210.html
suggested, but it did not work for me.
On Mon, Apr 11, 2011 at 7:50 PM, 茅旭峰 <m9...@gmail.com> wrote:
> Hi,
>
> Is it possible that some table cannot cover the whole key space. What I saw
> was like
>
> ====
> hbase(main):006:0> put 'table1', 'abc', 'cfEStore:dasd', '123'
>
> 0 row(s) in 0.3030 seconds
>
> hbase(main):007:0> put 'table1', 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
> 'cfEStore:dasd', '123'
>
> ERROR: java.io.IOException: HRegionInfo was null or empty in .META.,
> row=keyvalues={table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:server/1300672190144/Put/vlen=14,
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:serverstartcode/1300672190144/Put/vlen=8}
>
> Here is some help for this command:
> Put a cell 'value' at specified table/row/column and optionally
> timestamp coordinates. To put a cell value into table 't1' at
> row 'r1' under column 'c1' marked with the time 'ts1', do:
>
> hbase> put 't1', 'r1', 'c1', 'value', ts1
>
> ====
>
> I guess this means our table has lost some key range for the specific
> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
> is this reasonable? I wonder what might cause this issue? How can I check
> it?
>
> One weird thing, I can see such lines like
>
> ====
>
> table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,1300519432575.0bdd3d8fa7fc710860a4ee51fc9c8625.
> LC3MILeAUy8HmRFgU5-ESE-9T7w= LD4jOJWFyt4m7A3KGFST6d-uj3A=
> ====
>
> from the web UI, which I think means the table has holds the key range for
> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='
>
> I also have not seen any errors/warnings in the log file. Any suggestions?
>
> Thanks in advance.
>
Re: The whole key space is not covered by the .META.
Posted by 茅旭峰 <m9...@gmail.com>.
Or does this mean I've corrupted the .META. data? BTW, any way to recover
the .META.?
On Mon, Apr 11, 2011 at 7:50 PM, 茅旭峰 <m9...@gmail.com> wrote:
> Hi,
>
> Is it possible that some table cannot cover the whole key space. What I saw
> was like
>
> ====
> hbase(main):006:0> put 'table1', 'abc', 'cfEStore:dasd', '123'
>
> 0 row(s) in 0.3030 seconds
>
> hbase(main):007:0> put 'table1', 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
> 'cfEStore:dasd', '123'
>
> ERROR: java.io.IOException: HRegionInfo was null or empty in .META.,
> row=keyvalues={table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:server/1300672190144/Put/vlen=14,
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:serverstartcode/1300672190144/Put/vlen=8}
>
> Here is some help for this command:
> Put a cell 'value' at specified table/row/column and optionally
> timestamp coordinates. To put a cell value into table 't1' at
> row 'r1' under column 'c1' marked with the time 'ts1', do:
>
> hbase> put 't1', 'r1', 'c1', 'value', ts1
>
> ====
>
> I guess this means our table has lost some key range for the specific
> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
> is this reasonable? I wonder what might cause this issue? How can I check
> it?
>
> One weird thing, I can see such lines like
>
> ====
>
> table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,1300519432575.0bdd3d8fa7fc710860a4ee51fc9c8625.
> LC3MILeAUy8HmRFgU5-ESE-9T7w= LD4jOJWFyt4m7A3KGFST6d-uj3A=
> ====
>
> from the web UI, which I think means the table has holds the key range for
> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='
>
> I also have not seen any errors/warnings in the log file. Any suggestions?
>
> Thanks in advance.
>