You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Adam Hunt <ad...@gmail.com> on 2017/09/20 12:45:30 UTC
Region Stuck in "OPENING" state
I'm are running HBase 1.1.8 on MapR 5.2.1 and I have a table with a little
over 1000 regions. All the regions load fine, except for one, which is
stuck in the opening state. I've tried restarting the masters and all the
region servers, but I'm getting the same result. The logs don't seem very
helpful. They show that the request to open the region, but it never
succeeds.
2017-09-20 05:26:02,095 INFO
[PriorityRpcServer.handler=3,queue=1,port=16020]
regionserver.RSRpcServices: Open whois-versioned,bobandamy.net
,1505420182154.489a652d37429afe19955512f644f64f.
2017-09-20 05:26:02,103 INFO
[StoreOpener-489a652d37429afe19955512f644f64f-1] hfile.CacheConfig:
blockCache=LruBlockCache{blockCount=17212, currentSize=1451765224,
freeSize=4952805912, maxSize=6404571136, heapSize=1451765224,
minSize=6084342272, minFactor=0.95, multiSize=3042171136, multiFactor=0.5,
singleSize=1521085568, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false, cacheEvictOnClose=false,
cacheDataCompressed=false, prefetchOnOpen=false
2017-09-20 05:26:02,103 INFO
[StoreOpener-489a652d37429afe19955512f644f64f-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; major period 604800000, major jitter 0.500000, min locality to
compact 0.000000
2017-09-20 05:26:02,106 INFO
[StoreOpener-489a652d37429afe19955512f644f64f-1] hfile.CacheConfig:
blockCache=LruBlockCache{blockCount=17212, currentSize=1451765224,
freeSize=4952805912, maxSize=6404571136, heapSize=1451765224,
minSize=6084342272, minFactor=0.95, multiSize=3042171136, multiFactor=0.5,
singleSize=1521085568, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false, cacheEvictOnClose=false,
cacheDataCompressed=false, prefetchOnOpen=false
hbck reports this error:
ERROR: Region { meta =>
whois-versioned,bobandamy.net,1505420182154.489a652d37429afe19955512f644f64f.,
hdfs =>
maprfs:///hbase/data/default/whois-versioned/489a652d37429afe19955512f644f64f,
deployed => , replicaId => 0 } not deployed on any region server.
2017-09-20 05:29:10,602 INFO [main] util.HBaseFsck: Handling overlap
merges in parallel. set hbasefsck.overlap.merge.parallel to false to run
serially.
ERROR: There is a hole in the region chain between bobandamy.net and
bodyinbalanceshop.com. You need to create a new .regioninfo and region dir
in hdfs to plug the hole.
ERROR: Found inconsistency in table whois-versioned
Repairing the table created more of a mess, and didn't fix this issue. I've
tried the assigning it from the shell, but the operation just times out.
Any ideas on how to debug this issue in order to recover the data would be
greatly appreciated. Thank you.
Re: Region Stuck in "OPENING" state
Posted by Biju N <bi...@gmail.com>.
If you haven't tried already, use "hbase hbck" to fix the holes.
On Wed, Sep 20, 2017 at 8:45 AM, Adam Hunt <ad...@gmail.com> wrote:
> I'm are running HBase 1.1.8 on MapR 5.2.1 and I have a table with a little
> over 1000 regions. All the regions load fine, except for one, which is
> stuck in the opening state. I've tried restarting the masters and all the
> region servers, but I'm getting the same result. The logs don't seem very
> helpful. They show that the request to open the region, but it never
> succeeds.
>
> 2017-09-20 05:26:02,095 INFO
> [PriorityRpcServer.handler=3,queue=1,port=16020]
> regionserver.RSRpcServices: Open whois-versioned,bobandamy.net
> ,1505420182154.489a652d37429afe19955512f644f64f.
> 2017-09-20 05:26:02,103 INFO
> [StoreOpener-489a652d37429afe19955512f644f64f-1] hfile.CacheConfig:
> blockCache=LruBlockCache{blockCount=17212, currentSize=1451765224,
> freeSize=4952805912, maxSize=6404571136, heapSize=1451765224,
> minSize=6084342272, minFactor=0.95, multiSize=3042171136, multiFactor=0.5,
> singleSize=1521085568, singleFactor=0.25}, cacheDataOnRead=true,
> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
> cacheDataCompressed=false, prefetchOnOpen=false
> 2017-09-20 05:26:02,103 INFO
> [StoreOpener-489a652d37429afe19955512f644f64f-1]
> compactions.CompactionConfiguration: size [134217728,
> 9223372036854775807);
> files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
> 2684354560; major period 604800000, major jitter 0.500000, min locality to
> compact 0.000000
> 2017-09-20 05:26:02,106 INFO
> [StoreOpener-489a652d37429afe19955512f644f64f-1] hfile.CacheConfig:
> blockCache=LruBlockCache{blockCount=17212, currentSize=1451765224,
> freeSize=4952805912, maxSize=6404571136, heapSize=1451765224,
> minSize=6084342272, minFactor=0.95, multiSize=3042171136, multiFactor=0.5,
> singleSize=1521085568, singleFactor=0.25}, cacheDataOnRead=true,
> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
> cacheDataCompressed=false, prefetchOnOpen=false
>
> hbck reports this error:
>
> ERROR: Region { meta =>
> whois-versioned,bobandamy.net,1505420182154.489a652d37429afe19955512f644f6
> 4f.,
> hdfs =>
> maprfs:///hbase/data/default/whois-versioned/
> 489a652d37429afe19955512f644f64f,
> deployed => , replicaId => 0 } not deployed on any region server.
> 2017-09-20 05:29:10,602 INFO [main] util.HBaseFsck: Handling overlap
> merges in parallel. set hbasefsck.overlap.merge.parallel to false to run
> serially.
> ERROR: There is a hole in the region chain between bobandamy.net and
> bodyinbalanceshop.com. You need to create a new .regioninfo and region
> dir
> in hdfs to plug the hole.
> ERROR: Found inconsistency in table whois-versioned
>
> Repairing the table created more of a mess, and didn't fix this issue. I've
> tried the assigning it from the shell, but the operation just times out.
> Any ideas on how to debug this issue in order to recover the data would be
> greatly appreciated. Thank you.
>
Re: Region Stuck in "OPENING" state
Posted by Ted Yu <yu...@gmail.com>.
Can you turn on DEBUG logging ?
That may give you more idea on why the region doesn't open.
You can also use the http://hbase.apache.org/book.html#hfile_tool to
inspect hfiles in this region.
Cheers
On Wed, Sep 20, 2017 at 5:45 AM, Adam Hunt <ad...@gmail.com> wrote:
> I'm are running HBase 1.1.8 on MapR 5.2.1 and I have a table with a little
> over 1000 regions. All the regions load fine, except for one, which is
> stuck in the opening state. I've tried restarting the masters and all the
> region servers, but I'm getting the same result. The logs don't seem very
> helpful. They show that the request to open the region, but it never
> succeeds.
>
> 2017-09-20 05:26:02,095 INFO
> [PriorityRpcServer.handler=3,queue=1,port=16020]
> regionserver.RSRpcServices: Open whois-versioned,bobandamy.net
> ,1505420182154.489a652d37429afe19955512f644f64f.
> 2017-09-20 05:26:02,103 INFO
> [StoreOpener-489a652d37429afe19955512f644f64f-1] hfile.CacheConfig:
> blockCache=LruBlockCache{blockCount=17212, currentSize=1451765224,
> freeSize=4952805912, maxSize=6404571136, heapSize=1451765224,
> minSize=6084342272, minFactor=0.95, multiSize=3042171136, multiFactor=0.5,
> singleSize=1521085568, singleFactor=0.25}, cacheDataOnRead=true,
> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
> cacheDataCompressed=false, prefetchOnOpen=false
> 2017-09-20 05:26:02,103 INFO
> [StoreOpener-489a652d37429afe19955512f644f64f-1]
> compactions.CompactionConfiguration: size [134217728,
> 9223372036854775807);
> files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
> 2684354560; major period 604800000, major jitter 0.500000, min locality to
> compact 0.000000
> 2017-09-20 05:26:02,106 INFO
> [StoreOpener-489a652d37429afe19955512f644f64f-1] hfile.CacheConfig:
> blockCache=LruBlockCache{blockCount=17212, currentSize=1451765224,
> freeSize=4952805912, maxSize=6404571136, heapSize=1451765224,
> minSize=6084342272, minFactor=0.95, multiSize=3042171136, multiFactor=0.5,
> singleSize=1521085568, singleFactor=0.25}, cacheDataOnRead=true,
> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
> cacheDataCompressed=false, prefetchOnOpen=false
>
> hbck reports this error:
>
> ERROR: Region { meta =>
> whois-versioned,bobandamy.net,1505420182154.489a652d37429afe19955512f644f6
> 4f.,
> hdfs =>
> maprfs:///hbase/data/default/whois-versioned/
> 489a652d37429afe19955512f644f64f,
> deployed => , replicaId => 0 } not deployed on any region server.
> 2017-09-20 05:29:10,602 INFO [main] util.HBaseFsck: Handling overlap
> merges in parallel. set hbasefsck.overlap.merge.parallel to false to run
> serially.
> ERROR: There is a hole in the region chain between bobandamy.net and
> bodyinbalanceshop.com. You need to create a new .regioninfo and region
> dir
> in hdfs to plug the hole.
> ERROR: Found inconsistency in table whois-versioned
>
> Repairing the table created more of a mess, and didn't fix this issue. I've
> tried the assigning it from the shell, but the operation just times out.
> Any ideas on how to debug this issue in order to recover the data would be
> greatly appreciated. Thank you.
>