You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Michael Stack (Jira)" <ji...@apache.org> on 2020/03/05 15:28:00 UTC
[jira] [Comment Edited] (HBASE-23931) CatalogJanitor consistency
check adds merging regions to orphan list (tooo)
[ https://issues.apache.org/jira/browse/HBASE-23931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17052256#comment-17052256 ]
Michael Stack edited comment on HBASE-23931 at 3/5/20, 3:27 PM:
----------------------------------------------------------------
Let me reopen. This came of an upgrade from 2.1 to 2.3 with more to follow. Will see this again. Need to take a snapshot of meta pre-upgrade Here is storyline.
* On restart, region is assigned.
* CJ runs and reports overlap (Region an 'overlap' pre-upgrade?
* Run hbck2 fixMeta but it fails w/ below:
{{2020-03-04 23:40:43,874 ERROR org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure: Error trying to merge [8d4e9d9dbae0a90d47b9f72c89973693, 62dbb839b3a70d5b670b9a30b0a05648] in scooter_analytics_segment (in state=MERGE_TABLE_REGIONS_PREPARE)}}
{{org.apache.hadoop.hbase.exceptions.MergeRegionException: Skip merging [8d4e9d9dbae0a90d47b9f72c89973693, 62dbb839b3a70d5b670b9a30b0a05648], because parent [8d4e9d9dbae0a90d47b9f72c89973693] has a merge qualifier}}
{{ at org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.prepareMergeRegion(MergeTableRegionsProcedure.java:483)}}
{{ at org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:197)}}
{{ at org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:77)}}
{{ at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:194)}}
{{ at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:962)}}
{{ at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1669)}}
{{ at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1416)}}
{{ at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:79)}}
{{ at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1986)}}
was (Author: stack):
Let me reopen. This came of an upgrade from 2.1 to 2.3 with more to follow. Will see this again. Need to take a snapshot of meta pre-upgrade Here is storyline.
* On restart, region is assigned.
* CJ runs and reports overlap (Region an 'overlap' pre-upgrade?
* Run hbck2 fixMeta
* {{It schedules a merge but merge fails with MergeTableRegionsProcedure: Skip merging [8d4e9d9dbae0a90d47b9f72c89973693, 62dbb839b3a70d5b670b9a30b0a05648], because parent [8d4e9d9dbae0a90d47b9f72c89973693] has a merge qualifier}}
> CatalogJanitor consistency check adds merging regions to orphan list (tooo)
> ---------------------------------------------------------------------------
>
> Key: HBASE-23931
> URL: https://issues.apache.org/jira/browse/HBASE-23931
> Project: HBase
> Issue Type: Bug
> Components: hbck2
> Reporter: Michael Stack
> Priority: Major
>
> When the CJ runs, it does its consistency checking first before figuring if the current row is that of a splitting or merging region. Down in the guts of consistency we will check if its a splitting region and then cater appropriately accommodating daughters, etc. For merge we don't do the same. We'll actually add merge children parents as orphans. Its transitory but confusing.
>
> Here is an (ugly) example. Here is how the merge shows in orphans section of HBCK Report
> {code:java}
> {ENCODED => 8d4e9d9dbae0a90d47b9f72c89973693, NAME => 'scooter_analytics_segment,\x01\x03\x04\x01\x02\x03,1583362144713.8d4e9d9dbae0a90d47b9f72c89973693.', STARTKEY => '\x01\x03\x04\x01\x02\x03', ENDKEY => '\x01\x03\x04\x02\x02\x03'}{ENCODED => 62dbb839b3a70d5b670b9a30b0a05648, NAME => 'scooter_analytics_segment,\x01\x03\x04\x01\x02\x03\x02\x01\x01\x02201904090000\x00\x00qwest\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0DW:S:P:D\x03\x00\x00\x03,1554833869031.62dbb839b3a70d5b670b9a30b0a05648.', STARTKEY => '\x01\x03\x04\x01\x02\x03\x02\x01\x01\x02201904090000\x00\x00qwest\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0DW:S:P:D\x03\x00\x00\x03', ENDKEY => '\x01\x03\x04\x02\x01\x01'}{ENCODED => 8d4e9d9dbae0a90d47b9f72c89973693, NAME => 'scooter_analytics_segment,\x01\x03\x04\x01\x02\x03,1583362144713.8d4e9d9dbae0a90d47b9f72c89973693.', STARTKEY => '\x01\x03\x04\x01\x02\x03', ENDKEY => '\x01\x03\x04\x02\x02\x03'}{ENCODED => 2666b87ae4b58b05e815a3f14cc33215, NAME => 'scooter_analytics_segment,\x01\x03\x04\x02\x01\x01,1583365243857.2666b87ae4b58b05e815a3f14cc33215.', STARTKEY => '\x01\x03\x04\x02\x01\x01', ENDKEY => '\x01\x03\x04\x02\x02\x03'}{code}
> Then... here is the row in meta...
>
> {code:java}
> scooter_analytics_segment,\x01\x03\x04\x01\x02\x03,1583362144713.8d4e9d9dbae0a90d47b9f72c89973693. column=info:merge0000, timestamp=1583364483794, value=PBUF\x08\xE6\xE8\xFC\x9C\xA0-\x12$\x0A\x07default\x12\x19scooter_analytics_segment\x1A\x06\x01\x03\x04\x01\x02\x03"\x06\x01\x03\x04\x02\x02\x03(\x000\x008\x00B d097cacbd74c36e84a599926547e3249
> scooter_analytics_segment,\x01\x03\x04\x01\x02\x03,1583362144713.8d4e9d9dbae0a90d47b9f72c89973693. column=info:merge0001, timestamp=1583364483794, value=PBUF\x08\xC8\xF3\xF3\xBD\x8A.\x12$\x0A\x07default\x12\x19scooter_analytics_segment\x1A\x06\x01\x03\x04\x02\x01\x01"\x06\x01\x03\x04\x02\x02\x03(\x000\x008\x00B 6b2ac2d1878e688da50483014fc5ab58
> scooter_analytics_segment,\x01\x03\x04\x01\x02\x03,1583362144713.8d4e9d9dbae0a90d47b9f72c89973693. column=info:regioninfo, timestamp=1583364484283, value={ENCODED => 8d4e9d9dbae0a90d47b9f72c89973693, NAME => 'scooter_analytics_segment,\x01\x03\x04\x01\x02\x03,1583362144713.8d4e9d9dbae0a90d47b9f72c89973693.', STARTKEY => '\x01\x03\x04\x01\x02\x03', ENDKEY => '\x01\x03\x04\x02\x02\x03'}
> scooter_analytics_segment,\x01\x03\x04\x01\x02\x03,1583362144713.8d4e9d9dbae0a90d47b9f72c89973693. column=info:seqnumDuringOpen, timestamp=1583364484283, value=\x00\x00\x00\x00\x00\x00\x010
> scooter_analytics_segment,\x01\x03\x04\x01\x02\x03,1583362144713.8d4e9d9dbae0a90d47b9f72c89973693. column=info:server, timestamp=1583364484283, value=a.example.com:16020
> scooter_analytics_segment,\x01\x03\x04\x01\x02\x03,1583362144713.8d4e9d9dbae0a90d47b9f72c89973693. column=info:serverstartcode, timestamp=1583364484283, value=1583349446638
> scooter_analytics_segment,\x01\x03\x04\x01\x02\x03,1583362144713.8d4e9d9dbae0a90d47b9f72c89973693. column=info:sn, timestamp=1583364483948, value=a.example.com,16020,1583349446638
> scooter_analytics_segment,\x01\x03\x04\x01\x02\x03,1583362144713.8d4e9d9dbae0a90d47b9f72c89973693. column=info:state, timestamp=1583364484283, value=OPEN
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)