You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Zach Johnson <zj...@ccri.com> on 2020/12/11 14:36:53 UTC
Region stuck in transition Splitting
version: HBase 1.4.8
We have a stuck RIT that wont clear. The parent was offlined, and
daughter region were created and moved out of the parents .split dir.
But the parent region is never removed and the RIT doesn't clear. We
have tried to run `move '<parent_region_name>'`, `unassign
'<parent_region_name>'`, and |sudo stop hbase-regionserver && sudo start
hbase-regionserver |on the parent and daughter regionservers. My only
other thought is to disable the table, delete the parent region and
enable the table. Any advice on how to clear the RIT or insight on what
is causing it would be appreciated. I have included some logs bellow
that might be helpful.
Thanks,
Zach
Region server logs:
2020-12-04 14:00:35,430 INFO
[RpcServer.priority.FPBQ.Fifo.handler=17,queue=7,port=16020]
regionserver.RSRpcServices: Close d0b06e343bc8ae49ef7e5b66089fcb2d,
moving to null
2020-12-04 14:00:35,431 INFO
[StoreCloserThread-synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.-1]
regionserver.HStore: Closed d
2020-12-04 14:00:35,436 INFO [RS_CLOSE_REGION-ip-10-0-1-4:16020-0]
regionserver.HRegion: Closed
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
2020-12-04 14:00:35,438 INFO
[RpcServer.priority.FPBQ.Fifo.handler=17,queue=7,port=16020]
regionserver.RSRpcServices: Open
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
2020-12-04 14:00:35,703 INFO
[StoreOpener-d0b06e343bc8ae49ef7e5b66089fcb2d-1] hfile.CacheConfig:
Created cacheConfig for d:
blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@58ff4bf2,
cacheDataOnRead=true, cacheDataOnWrite=false,
cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2020-12-04 14:00:35,703 INFO
[StoreOpener-d0b06e343bc8ae49ef7e5b66089fcb2d-1]
compactions.CompactionConfiguration: size [1409286144, 5000000000,
5000000000); files [6, 10); ratio 1.000000; off-peak ratio 5.000000;
throttle point 28185722880; major period 0, major jitter 0.500000,
min locality to compact 0.000000; tiered compaction: max_age
9223372036854775807, incoming window min 6, compaction policy for
tiered window
org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
single output for minor true, compaction window factory
org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
2020-12-04 14:00:45,076 INFO [RS_OPEN_REGION-ip-10-0-1-4:16020-2]
regionserver.HRegion: Onlined d0b06e343bc8ae49ef7e5b66089fcb2d; next
sequenceid=2009960
2020-12-04 14:00:45,077 INFO
[PostOpenDeployTasks:d0b06e343bc8ae49ef7e5b66089fcb2d]
regionserver.HRegionServer: Post open deploy tasks for
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
2020-12-04 14:16:13,156 INFO
[RpcServer.priority.FPBQ.Fifo.handler=17,queue=7,port=16020]
regionserver.RSRpcServices: Splitting
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
2020-12-04 14:16:20,003 INFO
[regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
regionserver.SplitTransaction: Starting split of region
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
2020-12-04 14:16:20,457 INFO
[StoreCloserThread-synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.-1]
regionserver.HStore: Closed d
2020-12-04 14:16:20,462 INFO
[regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
regionserver.HRegion: Closed
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
2020-12-04 14:16:20,462 INFO
[regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
regionserver.SplitTransaction: Preparing to split 14 storefiles for
region
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
using 8 threads
2020-12-04 14:16:22,228 INFO
[regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
s3n2.S3NativeFileSystem2: rename
s3://bucket/root/data/default/synthetic_PRM_attr_dtg_v8_00609/d0b06e343bc8ae49ef7e5b66089fcb2d/.splits/453d941ff97a10bcf28e6ffe22d1d0e2
s3://bucket/root/data/default/synthetic_PRM_attr_dtg_v8_00609/453d941ff97a10bcf28e6ffe22d1d0e2
2020-12-04 14:16:23,988 INFO
[regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
s3n2.S3NativeFileSystem2: rename
s3://bucket/root/data/default/synthetic_PRM_attr_dtg_v8_00609/d0b06e343bc8ae49ef7e5b66089fcb2d/.splits/9d210ddbe593c83c62ac2d02413b7d97
s3://bucket/root/data/default/synthetic_PRM_attr_dtg_v8_00609/9d210ddbe593c83c62ac2d02413b7d97
2020-12-04 14:16:26,391 INFO
[regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
regionserver.SplitRequest: Running rollback/cleanup of failed split
of
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.;
Failed
ip-10-0-1-4.ec2.internal,16020,1606786380554-daughterOpener=9d210ddbe593c83c62ac2d02413b7d97
echo "scan 'hbase:meta'" grep file for region
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. column=info:regioninfo, timestamp=1607091385153, value={ENCODED => d0b06e343bc8ae49ef7e5b66089fcb2d, NAME => 'synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.', STARTKEY => '\x018000017536e25cf0\x00e13', ENDKEY => '\x02', OFFLINE => true, SPLIT => true}
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. column=info:seqnumDuringOpen, timestamp=1607090445077, value=\x00\x00\x00\x00\x00\x1E\xABh
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. column=info:server, timestamp=1607090445077, value=ip-10-0-1-4.ec2.internal:16020
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. column=info:serverstartcode, timestamp=1607090445077, value=1606786380554
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. column=info:sn, timestamp=1606786396087, value=ip-10-0-1-4.ec2.internal,16020,1606786380554
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. column=info:splitA, timestamp=1607091385153, value={ENCODED => 453d941ff97a10bcf28e6ffe22d1d0e2, NAME => 'synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1607091380003.453d941ff97a10bcf28e6ffe22d1d0e2.', STARTKEY => '\x018000017536e25cf0\x00e13', ENDKEY => '\x01800001753f422970\x0063'}
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. column=info:splitB, timestamp=1607091385153, value={ENCODED => 9d210ddbe593c83c62ac2d02413b7d97, NAME => 'synthetic_PRM_attr_dtg_v8_00609,\x01800001753f422970\x0063,1607091380003.9d210ddbe593c83c62ac2d02413b7d97.', STARTKEY => '\x01800001753f422970\x0063', ENDKEY => '\x02'}
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. column=info:state, timestamp=1607091380003, value=SPLITTING
Re: Region stuck in transition Splitting
Posted by Viraj Jasani <vj...@apache.org>.
Have you also tried restarting active master?
> regionserver.SplitRequest: Running rollback/cleanup of failed split
Can you provide few logs after this line?
Also, master logs for this region when it was going
through split will also be helpful.
> My only
> other thought is to disable the table, delete the parent region and
> enable the table.
This manual step should not be required.
Do you see any logs with RS_ZK_REQUEST_REGION_SPLIT on
master side?
Without further logs after "Running rollback/cleanup of failed split", I
am not sure if the issue is similar but you can try comparing logs
from https://issues.apache.org/jira/browse/HBASE-23261
If active master failover doesn't solve RIT, please upload further
logs as requested above on some public file hosting server and
you can provide link here. That will be easier way to provide
more logs than pasting all relevant logs over this thread.
On 2020/12/11 14:36:53, Zach Johnson <zj...@ccri.com> wrote:
> version: HBase 1.4.8
>
> We have a stuck RIT that wont clear. The parent was offlined, and
> daughter region were created and moved out of the parents .split dir.
> But the parent region is never removed and the RIT doesn't clear. We
> have tried to run `move '<parent_region_name>'`, `unassign
> '<parent_region_name>'`, and |sudo stop hbase-regionserver && sudo start
> hbase-regionserver |on the parent and daughter regionservers. My only
> other thought is to disable the table, delete the parent region and
> enable the table. Any advice on how to clear the RIT or insight on what
> is causing it would be appreciated. I have included some logs bellow
> that might be helpful.
>
> Thanks,
>
> Zach
>
> Region server logs:
>
> 2020-12-04 14:00:35,430 INFO
> [RpcServer.priority.FPBQ.Fifo.handler=17,queue=7,port=16020]
> regionserver.RSRpcServices: Close d0b06e343bc8ae49ef7e5b66089fcb2d,
> moving to null
> 2020-12-04 14:00:35,431 INFO
> [StoreCloserThread-synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.-1]
> regionserver.HStore: Closed d
> 2020-12-04 14:00:35,436 INFO [RS_CLOSE_REGION-ip-10-0-1-4:16020-0]
> regionserver.HRegion: Closed
> synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
> 2020-12-04 14:00:35,438 INFO
> [RpcServer.priority.FPBQ.Fifo.handler=17,queue=7,port=16020]
> regionserver.RSRpcServices: Open
> synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
> 2020-12-04 14:00:35,703 INFO
> [StoreOpener-d0b06e343bc8ae49ef7e5b66089fcb2d-1] hfile.CacheConfig:
> Created cacheConfig for d:
> blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@58ff4bf2,
> cacheDataOnRead=true, cacheDataOnWrite=false,
> cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
> cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
> 2020-12-04 14:00:35,703 INFO
> [StoreOpener-d0b06e343bc8ae49ef7e5b66089fcb2d-1]
> compactions.CompactionConfiguration: size [1409286144, 5000000000,
> 5000000000); files [6, 10); ratio 1.000000; off-peak ratio 5.000000;
> throttle point 28185722880; major period 0, major jitter 0.500000,
> min locality to compact 0.000000; tiered compaction: max_age
> 9223372036854775807, incoming window min 6, compaction policy for
> tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2020-12-04 14:00:45,076 INFO [RS_OPEN_REGION-ip-10-0-1-4:16020-2]
> regionserver.HRegion: Onlined d0b06e343bc8ae49ef7e5b66089fcb2d; next
> sequenceid=2009960
> 2020-12-04 14:00:45,077 INFO
> [PostOpenDeployTasks:d0b06e343bc8ae49ef7e5b66089fcb2d]
> regionserver.HRegionServer: Post open deploy tasks for
> synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
> 2020-12-04 14:16:13,156 INFO
> [RpcServer.priority.FPBQ.Fifo.handler=17,queue=7,port=16020]
> regionserver.RSRpcServices: Splitting
> synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
> 2020-12-04 14:16:20,003 INFO
> [regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
> regionserver.SplitTransaction: Starting split of region
> synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
> 2020-12-04 14:16:20,457 INFO
> [StoreCloserThread-synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.-1]
> regionserver.HStore: Closed d
> 2020-12-04 14:16:20,462 INFO
> [regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
> regionserver.HRegion: Closed
> synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
> 2020-12-04 14:16:20,462 INFO
> [regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
> regionserver.SplitTransaction: Preparing to split 14 storefiles for
> region
> synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
> using 8 threads
> 2020-12-04 14:16:22,228 INFO
> [regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
> s3n2.S3NativeFileSystem2: rename
> s3://bucket/root/data/default/synthetic_PRM_attr_dtg_v8_00609/d0b06e343bc8ae49ef7e5b66089fcb2d/.splits/453d941ff97a10bcf28e6ffe22d1d0e2
> s3://bucket/root/data/default/synthetic_PRM_attr_dtg_v8_00609/453d941ff97a10bcf28e6ffe22d1d0e2
> 2020-12-04 14:16:23,988 INFO
> [regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
> s3n2.S3NativeFileSystem2: rename
> s3://bucket/root/data/default/synthetic_PRM_attr_dtg_v8_00609/d0b06e343bc8ae49ef7e5b66089fcb2d/.splits/9d210ddbe593c83c62ac2d02413b7d97
> s3://bucket/root/data/default/synthetic_PRM_attr_dtg_v8_00609/9d210ddbe593c83c62ac2d02413b7d97
> 2020-12-04 14:16:26,391 INFO
> [regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
> regionserver.SplitRequest: Running rollback/cleanup of failed split
> of
> synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.;
> Failed
> ip-10-0-1-4.ec2.internal,16020,1606786380554-daughterOpener=9d210ddbe593c83c62ac2d02413b7d97
>
>
> echo "scan 'hbase:meta'" grep file for region
>
> synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. column=info:regioninfo, timestamp=1607091385153, value={ENCODED => d0b06e343bc8ae49ef7e5b66089fcb2d, NAME => 'synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.', STARTKEY => '\x018000017536e25cf0\x00e13', ENDKEY => '\x02', OFFLINE => true, SPLIT => true}
> synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. column=info:seqnumDuringOpen, timestamp=1607090445077, value=\x00\x00\x00\x00\x00\x1E\xABh
> synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. column=info:server, timestamp=1607090445077, value=ip-10-0-1-4.ec2.internal:16020
> synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. column=info:serverstartcode, timestamp=1607090445077, value=1606786380554
> synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. column=info:sn, timestamp=1606786396087, value=ip-10-0-1-4.ec2.internal,16020,1606786380554
> synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. column=info:splitA, timestamp=1607091385153, value={ENCODED => 453d941ff97a10bcf28e6ffe22d1d0e2, NAME => 'synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1607091380003.453d941ff97a10bcf28e6ffe22d1d0e2.', STARTKEY => '\x018000017536e25cf0\x00e13', ENDKEY => '\x01800001753f422970\x0063'}
> synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. column=info:splitB, timestamp=1607091385153, value={ENCODED => 9d210ddbe593c83c62ac2d02413b7d97, NAME => 'synthetic_PRM_attr_dtg_v8_00609,\x01800001753f422970\x0063,1607091380003.9d210ddbe593c83c62ac2d02413b7d97.', STARTKEY => '\x01800001753f422970\x0063', ENDKEY => '\x02'}
> synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. column=info:state, timestamp=1607091380003, value=SPLITTING
>
>