You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Austin Heyne <ah...@ccri.com> on 2019/07/05 14:25:26 UTC

Merged daughter region

Hey,

We're continuing to get some heavy use out of our normalizers (split 
point pr still to come), and we're seeing an issue with daughters 
getting merged. The situation is a region is splitting and becoming the 
parent to regions A and B, one of these regions, say B is then getting 
compacted out but region A is still waiting to be compacted. Region B is 
then getting merged by our normalization plan with it's neighbor region, 
C, producing region D. This leaves us with regions A and D and the 
metadata for the parent.

Now when a query comes in that spans A through D, the query planner sees 
the reference to B in the parent's metadata and tries to find that 
region. After the timeout period we get an exception that says the child 
region (B) of the parent isn't online yet but should be soon.

19/07/03 18:33:06 ERROR Executor: Exception in task 3.0 in stage 9.0 
(TID 43)
org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the 
location
     at 
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:316)
     at 
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
     at 
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
     at 
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:212)
     at 
org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:314)
     at 
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:289)
     at 
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:164)
     at 
org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:159)
     at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:796)

...
Caused by: org.apache.hadoop.hbase.client.RegionOfflineException: the 
only available region for the required row is a split parent, the 
daughters should be online soon: 
table_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f.
     at 
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1307)
     at 
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1183)
     at 
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)

 From what I gather we should be removing the reference to B from the 
parent's metadata when B gets merged with C.

Is this something people deal with or is there a good way to prevent this?

Thanks,
Austin


Re: Merged daughter region

Posted by Austin Heyne <ah...@ccri.com>.
Here's what our meta for this example looks like. I don't have the 
entries for the merged regions, aside from what is provided by the old 
parent. To try to prevent some confusion, in this example, it's region A 
that got compacted and merged (different from my initial example). Also 
this is 1.4.8 running on EMR and S3.

Parent:

banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f. 
column=info:regioninfo, timestamp=1561992899379, value={ENCODED => 
7b0d3c6836cf417b007771ab48d0450f, NAME => 
'banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f.', 
STARTKEY => '\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5', ENDKEY => 
'\x02\x09\xFC\x17\x9C\xC1\x01\xF0 n\xC3', OFFLINE => true, SPLIT => true}
  banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f. column=info:seqnumDuringOpen, timestamp=1561992267396, value=\x00\x00\x00\x00\x00\x12\xD59
  banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f. column=info:server, timestamp=1561992267396, value=ip-10-0-61-195.ec2.internal:16020
  banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f. column=info:serverstartcode, timestamp=1561992267396, value=1560961976157
  banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f. column=info:sn, timestamp=1560968683601, value=ip-10-0-61-195.ec2.internal,16020,1560961976157
  banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f. column=info:splitA, timestamp=1561992899379, value={ENCODED => a1d452e3efa13c9a8162a85b6ada9564, NAME => 'banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564.', STARTKEY => '\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5', ENDKEY => '\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8'}
  banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f. column=info:splitB, timestamp=1561992899379, value={ENCODED => 4046237c869f876fc8f9a5f090dc8060, NAME => 'banana-2556_banana_z3_v2,\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8,1561992895078.4046237c869f876fc8f9a5f090dc8060.', STARTKEY => '\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8', ENDKEY => '\x02\x09\xFC\x17\x9C\xC1\x01\xF0 n\xC3'}
  banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f. column=info:state, timestamp=1561992906651, value=SPLIT

Split A + previous region:

  banana-2556_banana_z3_v2,\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80,1561995018192.8152f86473b92265f7704d3ff9291f78. column=info:regioninfo, timestamp=1561995022370, value={ENCODED => 8152f86473b92265f7704d3ff9291f78, NAME => 'banana-2556_banana_z3_v2,\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80,15619
95018192.8152f86473b92265f7704d3ff9291f78.', STARTKEY => 
'\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80', ENDKEY => 
'\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8'}
  banana-2556_banana_z3_v2,\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80,1561995018192.8152f86473b92265f7704d3ff9291f78. column=info:seqnumDuringOpen, timestamp=1561995028121, value=\x00\x00\x00\x00\x00\x00\x00\x01
  banana-2556_banana_z3_v2,\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80,1561995018192.8152f86473b92265f7704d3ff9291f78. column=info:server, timestamp=1561995028121, value=ip-10-0-62-125.ec2.internal:16020
  banana-2556_banana_z3_v2,\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80,1561995018192.8152f86473b92265f7704d3ff9291f78. column=info:serverstartcode, timestamp=1561995028121, value=1561126540919
  banana-2556_banana_z3_v2,\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80,1561995018192.8152f86473b92265f7704d3ff9291f78. column=info:state, timestamp=1561995028121, value=OPEN

Split B:

banana-2556_banana_z3_v2,\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8,1561992895078.4046237c869f876fc8f9a5f090dc8060. 
column=info:regioninfo, timestamp=1561992899379, value={ENCODED => 
4046237c869f876fc8f9a5f090dc8060, NAME => 
'banana-2556_banana_z3_v2,\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8,1561992895078.4046237c869f876fc8f9a5f090dc8060.', 
STARTKEY => '\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8', ENDKEY => 
'\x02\x09\xFC\x17\x9C\xC1\x01\xF0 n\xC3'}
  banana-2556_banana_z3_v2,\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8,1561992895078.4046237c869f876fc8f9a5f090dc8060. column=info:seqnumDuringOpen, timestamp=1561992906652, value=\x00\x00\x00\x00\x00\x00\x00\x01
  banana-2556_banana_z3_v2,\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8,1561992895078.4046237c869f876fc8f9a5f090dc8060. column=info:server, timestamp=1561992906652, value=ip-10-0-61-195.ec2.internal:16020
  banana-2556_banana_z3_v2,\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8,1561992895078.4046237c869f876fc8f9a5f090dc8060. column=info:serverstartcode, timestamp=1561992906652, value=1560961976157
  banana-2556_banana_z3_v2,\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8,1561992895078.4046237c869f876fc8f9a5f090dc8060. column=info:state, timestamp=1561992906652, value=OPEN

You can see that the end key of "Split A + previous region" is the start 
key of Split B.

Looking at the logs from the operating regionserver we see the parent 
split (7b0d3c6836cf417b007771ab48d0450f):

2019-07-01 14:54:54,950 INFO 
[RpcServer.priority.FPBQ.Fifo.handler=19,queue=9,port=16020] 
regionserver.RSRpcServices: Splitting 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f.
2019-07-01 14:54:55,078 INFO 
[regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] 
regionserver.SplitTransaction: Starting split of region 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f.
2019-07-01 14:54:55,298 INFO 
[StoreCloserThread-banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f.-1] 
regionserver.HStore: Closed d
2019-07-01 14:54:55,303 INFO 
[regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] 
regionserver.HRegion: Closed 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f.
2019-07-01 14:54:55,303 INFO 
[regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] 
regionserver.SplitTransaction: Preparing to split 13 storefiles for 
region 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f. 
using 8 threads
2019-07-01 14:54:57,168 INFO 
[regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/.splits/a1d452e3efa13c9a8162a85b6ada9564 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564
2019-07-01 14:54:57,814 INFO 
[regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] 
regionserver.RegionCoprocessorHost: Loaded coprocessor 
org.locationtech.geomesa.hbase.coprocessor.GeoMesaCoprocessor from HTD 
of banana-2556_banana_z3_v2 successfully.
2019-07-01 14:54:58,406 INFO 
[regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/.splits/4046237c869f876fc8f9a5f090dc8060 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/4046237c869f876fc8f9a5f090dc8060
2019-07-01 14:54:59,166 INFO 
[regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] 
regionserver.RegionCoprocessorHost: Loaded coprocessor 
org.locationtech.geomesa.hbase.coprocessor.GeoMesaCoprocessor from HTD 
of banana-2556_banana_z3_v2 successfully.
...
2019-07-01 14:55:06,654 INFO 
[regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] 
regionserver.SplitRequest: Region split, hbase:meta updated, and report 
to master. 
Parent=banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f., 
new regions: 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564., 
banana-2556_banana_z3_v2,\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8,1561992895078.4046237c869f876fc8f9a5f090dc8060.. 
Split took 11sec
2019-07-01 14:55:06,623 INFO 
[ip-10-0-61-195.ec2.internal,16020,1560961976157-daughterOpener=a1d452e3efa13c9a8162a85b6ada9564] 
regionserver.HRegion: Onlined a1d452e3efa13c9a8162a85b6ada9564; next 
sequenceid=1234226
2019-07-01 14:55:06,651 INFO 
[ip-10-0-61-195.ec2.internal,16020,1560961976157-daughterOpener=4046237c869f876fc8f9a5f090dc8060] 
regionserver.HRegion: Onlined 4046237c869f876fc8f9a5f090dc8060; next 
sequenceid=1234227
2019-07-01 14:55:06,654 INFO 
[regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] 
regionserver.SplitRequest: Region split, hbase:meta updated, and report 
to master. 
Parent=banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f., 
new regions: 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564., 
banana-2556_banana_z3_v2,\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8,1561992895078.4046237c869f876fc8f9a5f090dc8060.. 
Split took 11sec
2019-07-01 14:55:06,654 INFO 
[regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] 
regionserver.SplitRequest: Split transaction journal:
     STARTED at 1561992895077
     PREPARED at 1561992895078
     BEFORE_PRE_SPLIT_HOOK at 1561992895078
     AFTER_PRE_SPLIT_HOOK at 1561992895078
     SET_SPLITTING at 1561992895079
     CREATE_SPLIT_DIR at 1561992895297
     CLOSED_PARENT_REGION at 1561992895303
     OFFLINED_PARENT at 1561992895303
     STARTED_REGION_A_CREATION at 1561992896718
     STARTED_REGION_B_CREATION at 1561992897974
     PONR at 1561992899379
     OPENED_REGION_A at 1561992906624
     OPENED_REGION_B at 1561992906651
     BEFORE_POST_SPLIT_HOOK at 1561992906653
     AFTER_POST_SPLIT_HOOK at 1561992906653
     COMPLETED at 1561992906653

Now for SplitA (a1d452e3efa13c9a8162a85b6ada9564):

2019-07-01 14:54:57,168 INFO 
[regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/.splits/a1d452e3efa13c9a8162a85b6ada9564 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564
2019-07-01 14:54:59,438 INFO 
[StoreOpener-a1d452e3efa13c9a8162a85b6ada9564-1] hfile.CacheConfig: 
Created cacheConfig for d: 
blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@93fa68a, 
cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, 
cacheBloomsOnWrite=false, cacheEvictOnClose=false, 
cacheDataCompressed=false, prefetchOnOpen=false
2019-07-01 14:54:59,438 INFO 
[StoreOpener-a1d452e3efa13c9a8162a85b6ada9564-1] 
compactions.CompactionConfiguration: size [1409286144, 5000000000, 
5000000000); files [6, 10); ratio 1.000000; off-peak ratio 5.000000; 
throttle point 28185722880; major period 0, major jitter 0.500000, min 
locality to compact 0.000000; tiered compaction: max_age 
9223372036854775807, incoming window min 6, compaction policy for tiered 
window 
org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, 
single output for minor true, compaction window factory 
org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
2019-07-01 15:15:18,542 INFO 
[RpcServer.priority.FPBQ.Fifo.handler=19,queue=9,port=16020] 
regionserver.RSRpcServices: Close a1d452e3efa13c9a8162a85b6ada9564, 
moving to ip-10-0-39-221.ec2.internal,16020,1560961973442
2019-07-01 15:15:18,544 INFO 
[StoreCloserThread-banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564.-1] 
regionserver.HStore: Closed d
2019-07-01 15:15:18,556 INFO [RS_CLOSE_REGION-ip-10-0-61-195:16020-1] 
regionserver.HRegion: Closed 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564.
2019-07-01 15:15:18,556 INFO [RS_CLOSE_REGION-ip-10-0-61-195:16020-1] 
regionserver.HRegionServer: Adding moved region record: 
a1d452e3efa13c9a8162a85b6ada9564 to 
ip-10-0-39-221.ec2.internal,16020,1560961973442 as of 1234226

<moved servers>

2019-07-01 15:15:18,566 INFO 
[RpcServer.priority.FPBQ.Fifo.handler=19,queue=9,port=16020] 
regionserver.RSRpcServices: Open 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564.
2019-07-01 15:15:18,675 INFO [RS_OPEN_REGION-ip-10-0-39-221:16020-0] 
regionserver.RegionCoprocessorHost: Loaded coprocessor 
org.locationtech.geomesa.hbase.coprocessor.GeoMesaCoprocessor from HTD 
of banana-2556_banana_z3_v2 successfully.
2019-07-01 15:15:18,781 INFO 
[StoreOpener-a1d452e3efa13c9a8162a85b6ada9564-1] hfile.CacheConfig: 
Created cacheConfig for d: 
blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@35b813db, 
cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, 
cacheBloomsOnWrite=false, cacheEvictOnClose=false, 
cacheDataCompressed=false, prefetchOnOpen=false
2019-07-01 15:15:18,782 INFO 
[StoreOpener-a1d452e3efa13c9a8162a85b6ada9564-1] 
compactions.CompactionConfiguration: size [1409286144, 5000000000, 
5000000000); files [6, 10); ratio 1.000000; off-peak ratio 5.000000; 
throttle point 28185722880; major period 0, major jitter 0.500000, min 
locality to compact 0.000000; tiered compaction: max_age 
9223372036854775807, incoming window min 6, compaction policy for tiered 
window 
org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, 
single output for minor true, compaction window factory 
org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
2019-07-01 15:15:21,266 INFO  [StoreFileOpenerThread-d-1] 
regionserver.StoreFile: Bloom filter turned off by CF config for 
2743f276b09d4a7e9bfcbef5f210c444_SeqId_4_.7b0d3c6836cf417b007771ab48d0450f
2019-07-01 15:15:28,018 INFO [RS_OPEN_REGION-ip-10-0-39-221:16020-0] 
regionserver.HRegion: Onlined a1d452e3efa13c9a8162a85b6ada9564; next 
sequenceid=1234230
2019-07-01 15:15:28,025 INFO 
[PostOpenDeployTasks:a1d452e3efa13c9a8162a85b6ada9564] 
regionserver.HRegionServer: Post open deploy tasks for 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564.
2019-07-01 15:15:28,025 INFO 
[regionserver/ip-10-0-39-221.ec2.internal/10.0.39.221:16020-longCompactions-1560968456884] 
regionserver.HRegion: Starting compaction on d in region 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564.
2019-07-01 15:15:28,025 INFO 
[regionserver/ip-10-0-39-221.ec2.internal/10.0.39.221:16020-longCompactions-1560968456884] 
regionserver.HStore: Starting compaction of 10 file(s) in d of 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564. 
into 
tmpdir=s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/.tmp, 
totalSize=30.8 G
2019-07-01 15:18:40,376 INFO 
[regionserver/ip-10-0-39-221.ec2.internal/10.0.39.221:16020-longCompactions-1560968456884] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/.tmp/f36978da3bd0413a9b3a57f51d00bb9b 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/f36978da3bd0413a9b3a57f51d00bb9b
2019-07-01 15:18:52,125 INFO 
[regionserver/ip-10-0-39-221.ec2.internal/10.0.39.221:16020-longCompactions-1560968456884] 
regionserver.HStore: Completed compaction of 10 file(s) in d of 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564. 
into f36978da3bd0413a9b3a57f51d00bb9b(size=7.9 G), total size for store 
is 9.5 G. This selection was in queue for 0sec, and took 3mins, 24sec to 
execute.
2019-07-01 15:18:52,125 INFO 
[regionserver/ip-10-0-39-221.ec2.internal/10.0.39.221:16020-longCompactions-1560968456884] 
regionserver.CompactSplitThread: Completed compaction: Request = 
regionName=banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564., 
storeName=d, fileCount=10, fileSize=30.8 G (9.3 G, 3.2 G, 3.2 G, 3.4 G, 
4.1 G, 677.2 M, 674.4 M, 4.7 G, 9.0 M, 16.0 M), priority=35, 
time=1818008526138670; duration=3mins, 24sec
2019-07-01 15:18:59,295 INFO 
[RS_COMPACTED_FILES_DISCHARGER-ip-10-0-39-221:16020-4] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/2743f276b09d4a7e9bfcbef5f210c444_SeqId_4_.7b0d3c6836cf417b007771ab48d0450f 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/2743f276b09d4a7e9bfcbef5f210c444_SeqId_4_.7b0d3c6836cf417b007771ab48d0450f
2019-07-01 15:18:59,648 INFO 
[RS_COMPACTED_FILES_DISCHARGER-ip-10-0-39-221:16020-4] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/d63e5131cdf5475e904a39f771c459e9.7b0d3c6836cf417b007771ab48d0450f 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/d63e5131cdf5475e904a39f771c459e9.7b0d3c6836cf417b007771ab48d0450f
2019-07-01 15:19:00,042 INFO 
[RS_COMPACTED_FILES_DISCHARGER-ip-10-0-39-221:16020-4] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/864b1dbe8fcd43d2aaaf573944dea7e7.7b0d3c6836cf417b007771ab48d0450f 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/864b1dbe8fcd43d2aaaf573944dea7e7.7b0d3c6836cf417b007771ab48d0450f
2019-07-01 15:19:00,405 INFO 
[RS_COMPACTED_FILES_DISCHARGER-ip-10-0-39-221:16020-4] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/06cfc7719e644cbf9a3a0ffb844aef7f.7b0d3c6836cf417b007771ab48d0450f 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/06cfc7719e644cbf9a3a0ffb844aef7f.7b0d3c6836cf417b007771ab48d0450f
2019-07-01 15:19:00,748 INFO 
[RS_COMPACTED_FILES_DISCHARGER-ip-10-0-39-221:16020-4] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/be1a9bee31c44afc9ae0e8a6c8af2053.7b0d3c6836cf417b007771ab48d0450f 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/be1a9bee31c44afc9ae0e8a6c8af2053.7b0d3c6836cf417b007771ab48d0450f
2019-07-01 15:19:01,194 INFO 
[RS_COMPACTED_FILES_DISCHARGER-ip-10-0-39-221:16020-4] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/322040d1fb4343f0abb91c9592433288.7b0d3c6836cf417b007771ab48d0450f 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/322040d1fb4343f0abb91c9592433288.7b0d3c6836cf417b007771ab48d0450f
2019-07-01 15:19:01,517 INFO 
[RS_COMPACTED_FILES_DISCHARGER-ip-10-0-39-221:16020-4] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/7bf98e832ff743edae58bbcab60bad75.7b0d3c6836cf417b007771ab48d0450f 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/7bf98e832ff743edae58bbcab60bad75.7b0d3c6836cf417b007771ab48d0450f
2019-07-01 15:19:02,008 INFO 
[RS_COMPACTED_FILES_DISCHARGER-ip-10-0-39-221:16020-4] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/be4c74dd039245f4ad1d57b768126501.7b0d3c6836cf417b007771ab48d0450f 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/be4c74dd039245f4ad1d57b768126501.7b0d3c6836cf417b007771ab48d0450f
2019-07-01 15:19:02,422 INFO 
[RS_COMPACTED_FILES_DISCHARGER-ip-10-0-39-221:16020-4] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/356197dbd10f4bb684af39152c637b5e.7b0d3c6836cf417b007771ab48d0450f 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/356197dbd10f4bb684af39152c637b5e.7b0d3c6836cf417b007771ab48d0450f
2019-07-01 15:19:02,893 INFO 
[RS_COMPACTED_FILES_DISCHARGER-ip-10-0-39-221:16020-4] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/be22002763d14728b151524afd376fd7.7b0d3c6836cf417b007771ab48d0450f 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/be22002763d14728b151524afd376fd7.7b0d3c6836cf417b007771ab48d0450f
2019-07-01 15:22:43,326 INFO 
[RpcServer.priority.FPBQ.Fifo.handler=19,queue=9,port=16020] 
regionserver.RSRpcServices: Close a1d452e3efa13c9a8162a85b6ada9564, 
moving to ip-10-0-62-125.ec2.internal,16020,1561126540919
2019-07-01 15:22:43,327 INFO 
[StoreCloserThread-banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564.-1] 
regionserver.HStore: Closed d
2019-07-01 15:22:43,338 INFO [RS_CLOSE_REGION-ip-10-0-39-221:16020-1] 
regionserver.HRegion: Closed 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564.
2019-07-01 15:22:43,338 INFO [RS_CLOSE_REGION-ip-10-0-39-221:16020-1] 
regionserver.HRegionServer: Adding moved region record: 
a1d452e3efa13c9a8162a85b6ada9564 to 
ip-10-0-62-125.ec2.internal,16020,1561126540919 as of 1234230

<moved servers>

2019-07-01 15:22:43,340 INFO 
[RpcServer.priority.FPBQ.Fifo.handler=18,queue=8,port=16020] 
regionserver.RSRpcServices: Open 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564.
2019-07-01 15:22:43,535 INFO 
[StoreOpener-a1d452e3efa13c9a8162a85b6ada9564-1] hfile.CacheConfig: 
Created cacheConfig for d: 
blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2581002f, 
cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, 
cacheBloomsOnWrite=false, cacheEvictOnClose=false, 
cacheDataCompressed=false, prefetchOnOpen=false
2019-07-01 15:22:43,536 INFO 
[StoreOpener-a1d452e3efa13c9a8162a85b6ada9564-1] 
compactions.CompactionConfiguration: size [1409286144, 5000000000, 
5000000000); files [6, 10); ratio 1.000000; off-peak ratio 5.000000; 
throttle point 28185722880; major period 0, major jitter 0.500000, min 
locality to compact 0.000000; tiered compaction: max_age 
9223372036854775807, incoming window min 6, compaction policy for tiered 
window 
org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, 
single output for minor true, compaction window factory 
org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
2019-07-01 15:22:46,574 INFO [RS_OPEN_REGION-ip-10-0-62-125:16020-7] 
regionserver.HRegion: Onlined a1d452e3efa13c9a8162a85b6ada9564; next 
sequenceid=1234234
2019-07-01 15:22:46,575 INFO 
[PostOpenDeployTasks:a1d452e3efa13c9a8162a85b6ada9564] 
regionserver.HRegionServer: Post open deploy tasks for 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564.
2019-07-01 15:22:46,576 INFO 
[regionserver/ip-10-0-62-125.ec2.internal/10.0.62.125:16020-shortCompactions-1561136546586] 
regionserver.HRegion: Starting compaction on d in region 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564.
2019-07-01 15:22:46,576 INFO 
[regionserver/ip-10-0-62-125.ec2.internal/10.0.62.125:16020-shortCompactions-1561136546586] 
regionserver.HStore: Starting compaction of 3 file(s) in d of 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564. 
into 
tmpdir=s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/.tmp, 
totalSize=1.6 G
2019-07-01 15:22:46,587 INFO 
[RpcServer.priority.FPBQ.Fifo.handler=18,queue=8,port=16020] 
regionserver.RSRpcServices: Receiving merging request for 
banana-2556_banana_z3_v2,\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80,1550014268044.8a99df0ca2c097ee707bc1eeb7402745., 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564.,forcible=true
2019-07-01 15:22:52,511 INFO 
[regionserver/ip-10-0-62-125.ec2.internal/10.0.62.125:16020-shortCompactions-1561136546586] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/.tmp/57d4d0aa4e1e437ca5ff8dbcc953a070 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/57d4d0aa4e1e437ca5ff8dbcc953a070
2019-07-01 15:22:54,458 INFO 
[regionserver/ip-10-0-62-125.ec2.internal/10.0.62.125:16020-shortCompactions-1561136546586] 
regionserver.HStore: Completed compaction of 3 file(s) in d of 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564. 
into 57d4d0aa4e1e437ca5ff8dbcc953a070(size=78.1 M), total size for store 
is 8.0 G. This selection was in queue for 0sec, and took 7sec to execute.
2019-07-01 15:22:54,458 INFO 
[regionserver/ip-10-0-62-125.ec2.internal/10.0.62.125:16020-shortCompactions-1561136546586] 
regionserver.CompactSplitThread: Completed compaction: Request = 
regionName=banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564., 
storeName=d, fileCount=3, fileSize=1.6 G (676.8 M, 677.1 M, 291.4 M), 
priority=44, time=868204396698301; duration=7sec
2019-07-01 15:23:45,642 INFO 
[RS_COMPACTED_FILES_DISCHARGER-ip-10-0-62-125:16020-5] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/25053cc8175047ed94d9acae9bb43a32.7b0d3c6836cf417b007771ab48d0450f 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/25053cc8175047ed94d9acae9bb43a32.7b0d3c6836cf417b007771ab48d0450f
2019-07-01 15:23:46,529 INFO 
[RS_COMPACTED_FILES_DISCHARGER-ip-10-0-62-125:16020-5] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/2834cd5210dd41e0ae7a523b276bf040.7b0d3c6836cf417b007771ab48d0450f 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/2834cd5210dd41e0ae7a523b276bf040.7b0d3c6836cf417b007771ab48d0450f
2019-07-01 15:23:46,989 INFO 
[RS_COMPACTED_FILES_DISCHARGER-ip-10-0-62-125:16020-5] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/cd61f91a73574edeb1c56f0fd50908db.7b0d3c6836cf417b007771ab48d0450f 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/cd61f91a73574edeb1c56f0fd50908db.7b0d3c6836cf417b007771ab48d0450f
2019-07-01 15:30:18,099 INFO 
[RpcServer.priority.FPBQ.Fifo.handler=18,queue=8,port=16020] 
regionserver.RSRpcServices: Receiving merging request for 
banana-2556_banana_z3_v2,\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80,1550014268044.8a99df0ca2c097ee707bc1eeb7402745., 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564.,forcible=true
2019-07-01 15:30:18,192 INFO 
[regionserver/ip-10-0-62-125.ec2.internal/10.0.62.125:16020-merges-1561472619525] 
regionserver.RegionMergeTransactionImpl: Starting merge of 
banana-2556_banana_z3_v2,\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80,1550014268044.8a99df0ca2c097ee707bc1eeb7402745. 
and 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564., 
forcible=true
2019-07-01 15:30:18,472 INFO 
[StoreCloserThread-banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564.-1] 
regionserver.HStore: Closed d
2019-07-01 15:30:18,476 INFO 
[regionserver/ip-10-0-62-125.ec2.internal/10.0.62.125:16020-merges-1561472619525] 
regionserver.HRegion: Closed 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564.
2019-07-01 15:30:28,124 INFO 
[regionserver/ip-10-0-62-125.ec2.internal/10.0.62.125:16020-merges-1561472619525] 
regionserver.RegionMergeRequest: Regions merged, hbase:meta updated, and 
report to master. 
region_a=banana-2556_banana_z3_v2,\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80,1550014268044.8a99df0ca2c097ee707bc1eeb7402745., 
region_b=banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564.,merged 
region=banana-2556_banana_z3_v2,\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80,1561995018192.8152f86473b92265f7704d3ff9291f78.. 
Region merge took 10sec

And SplitB (4046237c869f876fc8f9a5f090dc8060):

2019-07-01 14:54:58,406 INFO 
[regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/.splits/4046237c869f876fc8f9a5f090dc8060 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/4046237c869f876fc8f9a5f090dc8060
2019-07-01 14:54:59,431 INFO 
[StoreOpener-4046237c869f876fc8f9a5f090dc8060-1] hfile.CacheConfig: 
Created cacheConfig for d: 
blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@93fa68a, 
cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, 
cacheBloomsOnWrite=false, cacheEvictOnClose=false, 
cacheDataCompressed=false, prefetchOnOpen=false
2019-07-01 14:54:59,431 INFO 
[StoreOpener-4046237c869f876fc8f9a5f090dc8060-1] 
compactions.CompactionConfiguration: size [1409286144, 5000000000, 
5000000000); files [6, 10); ratio 1.000000; off-peak ratio 5.000000; 
throttle point 28185722880; major period 0, major jitter 0.500000, min 
locality to compact 0.000000; tiered compaction: max_age 
9223372036854775807, incoming window min 6, compaction policy for tiered 
window 
org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, 
single output for minor true, compaction window factory 
org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
2019-07-01 14:55:06,651 INFO 
[ip-10-0-61-195.ec2.internal,16020,1560961976157-daughterOpener=4046237c869f876fc8f9a5f090dc8060] 
regionserver.HRegion: Onlined 4046237c869f876fc8f9a5f090dc8060; next 
sequenceid=1234227
2019-07-01 14:55:06,654 INFO 
[regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] 
regionserver.SplitRequest: Region split, hbase:meta updated, and report 
to master. 
Parent=banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f., 
new regions: 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564., 
banana-2556_banana_z3_v2,\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8,1561992895078.4046237c869f876fc8f9a5f090dc8060.. 
Split took 11sec

There is then no activity around SplitB or the parent until I kicked off 
a major compaction a few days later. After which we then saw the parent 
being removed:

hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 
09:52:21,960 INFO 
[ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/06cfc7719e644cbf9a3a0ffb844aef7f 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/06cfc7719e644cbf9a3a0ffb844aef7f
hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 
09:52:32,028 INFO 
[ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/25053cc8175047ed94d9acae9bb43a32 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/25053cc8175047ed94d9acae9bb43a32
hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 
09:52:35,875 INFO 
[ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/2743f276b09d4a7e9bfcbef5f210c444_SeqId_4_ 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/2743f276b09d4a7e9bfcbef5f210c444_SeqId_4_
hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 
09:52:55,075 INFO 
[ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/2834cd5210dd41e0ae7a523b276bf040 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/2834cd5210dd41e0ae7a523b276bf040
hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 
09:52:58,668 INFO 
[ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/322040d1fb4343f0abb91c9592433288 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/322040d1fb4343f0abb91c9592433288
hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 
09:53:01,337 INFO 
[ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/356197dbd10f4bb684af39152c637b5e 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/356197dbd10f4bb684af39152c637b5e
hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 
09:53:02,125 INFO 
[ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/7bf98e832ff743edae58bbcab60bad75 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/7bf98e832ff743edae58bbcab60bad75
hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 
09:53:15,254 INFO 
[ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/864b1dbe8fcd43d2aaaf573944dea7e7 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/864b1dbe8fcd43d2aaaf573944dea7e7
hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 
09:53:21,139 INFO 
[ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/be1a9bee31c44afc9ae0e8a6c8af2053 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/be1a9bee31c44afc9ae0e8a6c8af2053
hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 
09:53:30,645 INFO 
[ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/be22002763d14728b151524afd376fd7 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/be22002763d14728b151524afd376fd7
hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 
09:53:31,515 INFO 
[ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/be4c74dd039245f4ad1d57b768126501 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/be4c74dd039245f4ad1d57b768126501
hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 
09:53:40,128 INFO 
[ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/cd61f91a73574edeb1c56f0fd50908db 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/cd61f91a73574edeb1c56f0fd50908db
hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 
09:53:44,638 INFO 
[ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] 
s3n2.S3NativeFileSystem2: rename 
s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/d63e5131cdf5475e904a39f771c459e9 
s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/d63e5131cdf5475e904a39f771c459e9
hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 
09:53:51,974 INFO 
[ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] 
hbase.MetaTableAccessor: Deleted 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f.

After this compaction queries to this key space began work again.

Thanks for the help,
Austin

On 7/5/19 10:44 AM, Wellington Chevreuil wrote:
> Hi Austin, which hbase version is in use? Is this
> '7b0d3c6836cf417b007771ab48d0450f' the parent region, or one of the
> daughter regions that got merged? What does meta scan show for the parent
> region and original two daughters, plus the resulting region from the merge?
>
> Em sex, 5 de jul de 2019 às 15:25, Austin Heyne <ah...@ccri.com> escreveu:
>
>> Hey,
>>
>> We're continuing to get some heavy use out of our normalizers (split
>> point pr still to come), and we're seeing an issue with daughters
>> getting merged. The situation is a region is splitting and becoming the
>> parent to regions A and B, one of these regions, say B is then getting
>> compacted out but region A is still waiting to be compacted. Region B is
>> then getting merged by our normalization plan with it's neighbor region,
>> C, producing region D. This leaves us with regions A and D and the
>> metadata for the parent.
>>
>> Now when a query comes in that spans A through D, the query planner sees
>> the reference to B in the parent's metadata and tries to find that
>> region. After the timeout period we get an exception that says the child
>> region (B) of the parent isn't online yet but should be soon.
>>
>> 19/07/03 18:33:06 ERROR Executor: Exception in task 3.0 in stage 9.0
>> (TID 43)
>> org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the
>> location
>>       at
>>
>> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:316)
>>       at
>>
>> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
>>       at
>>
>> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
>>       at
>>
>> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:212)
>>       at
>> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:314)
>>       at
>>
>> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:289)
>>       at
>>
>> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:164)
>>       at
>> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:159)
>>       at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:796)
>>
>> ...
>> Caused by: org.apache.hadoop.hbase.client.RegionOfflineException: the
>> only available region for the required row is a split parent, the
>> daughters should be online soon:
>>
>> table_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f.
>>       at
>>
>> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1307)
>>       at
>>
>> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1183)
>>       at
>>
>> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
>>
>>   From what I gather we should be removing the reference to B from the
>> parent's metadata when B gets merged with C.
>>
>> Is this something people deal with or is there a good way to prevent this?
>>
>> Thanks,
>> Austin
>>
>>



Re: Merged daughter region

Posted by Wellington Chevreuil <we...@gmail.com>.
Hi Austin, which hbase version is in use? Is this
'7b0d3c6836cf417b007771ab48d0450f' the parent region, or one of the
daughter regions that got merged? What does meta scan show for the parent
region and original two daughters, plus the resulting region from the merge?

Em sex, 5 de jul de 2019 às 15:25, Austin Heyne <ah...@ccri.com> escreveu:

> Hey,
>
> We're continuing to get some heavy use out of our normalizers (split
> point pr still to come), and we're seeing an issue with daughters
> getting merged. The situation is a region is splitting and becoming the
> parent to regions A and B, one of these regions, say B is then getting
> compacted out but region A is still waiting to be compacted. Region B is
> then getting merged by our normalization plan with it's neighbor region,
> C, producing region D. This leaves us with regions A and D and the
> metadata for the parent.
>
> Now when a query comes in that spans A through D, the query planner sees
> the reference to B in the parent's metadata and tries to find that
> region. After the timeout period we get an exception that says the child
> region (B) of the parent isn't online yet but should be soon.
>
> 19/07/03 18:33:06 ERROR Executor: Exception in task 3.0 in stage 9.0
> (TID 43)
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the
> location
>      at
>
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:316)
>      at
>
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
>      at
>
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
>      at
>
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:212)
>      at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:314)
>      at
>
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:289)
>      at
>
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:164)
>      at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:159)
>      at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:796)
>
> ...
> Caused by: org.apache.hadoop.hbase.client.RegionOfflineException: the
> only available region for the required row is a split parent, the
> daughters should be online soon:
>
> table_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f.
>      at
>
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1307)
>      at
>
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1183)
>      at
>
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
>
>  From what I gather we should be removing the reference to B from the
> parent's metadata when B gets merged with C.
>
> Is this something people deal with or is there a good way to prevent this?
>
> Thanks,
> Austin
>
>