You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Michael Stack (Jira)" <ji...@apache.org> on 2020/05/04 15:29:00 UTC
[jira] [Resolved] (HBASE-24246) Miscellaneous hbck2 fixMeta bulk
merge fixes: better logging around merges/overlap-fixing, 'HBCK Report'
overlap listing, and configuration
[ https://issues.apache.org/jira/browse/HBASE-24246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Stack resolved HBASE-24246.
-----------------------------------
Hadoop Flags: Reviewed
Assignee: Michael Stack
Resolution: Fixed
Pushed to branch-2.3+
Thanks for reviews [~psomogyi] and [~liuml07]
> Miscellaneous hbck2 fixMeta bulk merge fixes: better logging around merges/overlap-fixing, 'HBCK Report' overlap listing, and configuration
> -------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-24246
> URL: https://issues.apache.org/jira/browse/HBASE-24246
> Project: HBase
> Issue Type: Bug
> Reporter: Michael Stack
> Assignee: Michael Stack
> Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
> Attachments: Screen Shot 2020-04-23 at 4.42.41 PM.png
>
>
> Bit of a grab bag of small items.
> Improve log messages when bulk merge runs (particularly out of 'hbck2 fixMeta').
> Change config so we up the merge maximum since current low max is hit easily. When its hit, its merge of merge of merges w/ confusing output while happening. LOG when we hit max!
> {code}
> private static final String MAX_MERGE_COUNT_KEY = "hbase.master.metafixer.max.merge.count";
> private static final int MAX_MERGE_COUNT_DEFAULT = 10;
> {code}
> Fix the 'HBCK Report' page so it only logs encoded region names because if region name is large, the UI becomes unreadable.
> Fixup some logging messages. As is, can confuse operator.
> E.g. explain why having a merge qualifier blocks a region being merged up
> {code}
> org.apache.hadoop.hbase.exceptions.MergeRegionException: Skip merging [e1ded5983cb00c1a1a45e17414c5914c, 2634c71a10cf4a216b7c1608b7d24e6a, e7d40bb7ab50cdeb20118043aedf2cdc, a9ea228464bd8a40b8908cd263e2f6be, 544d32880131443eebb2a01691994b0d, a82d462a6edca634e4f8ecff4e8bc55c, 03e30d277a1ab977f93c103cb917c7d7, d0d5fb364838b00942448b89b3d802ff, 00ba78a9be4b27cf076622f49c45a43e, 338efcfc48928815423d7a6601b9e477], because parent [e1ded5983cb00c1a1a45e17414c5914c] has a merge qualifier
> {code}
> Add note to orphans page that you shouldn't try to address orphans until all meta issues are fixed (overlaps, holes).
> Add example on how to fix orphans with bulkload to the orphans section in hbck report. See HBASE-23928.
> Why the angle brackets in here:
> {code}
> Why angle brackets here? Usually for lists.
> 2020-04-23 15:21:13,471 INFO org.apache.hadoop.hbase.master.HMaster: Client=hbase//10.103.230.91 merge regions [e1ded5983cb00c1a1a45e17414c5914c], [2634c71a10cf4a216b7c1608b7d24e6a], [e7d40bb7ab50cdeb20118043aedf2cdc], [a9ea228464bd8a40b8908cd263e2f6be], [544d32880131443eebb2a01691994b0d], [a82d462a6edca634e4f8ecff4e8bc55c], [03e30d277a1ab977f93c103cb917c7d7], [d0d5fb364838b00942448b89b3d802ff], [50cbee16764bb09a9887a07d3a01304f], [338efcfc48928815423d7a6601b9e477]
> {code}
> These logs make no sense... need region name at least
> {code}
> What is this? Missing region
> Whats it doing?
> 2020-04-23 15:36:19,896 WARN org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress: totalCompactingKVs=500851692 less than currentCompactedKVs=562814154
> 2020-04-23 15:36:22,953 WARN org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress: totalCompactingKVs=500851692 less than currentCompactedKVs=564904980
> 2020-04-23 15:36:26,011 WARN org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress: totalCompactingKVs=500851692 less than currentCompactedKVs=566948397
> 2020-04-23 15:36:29,071 WARN org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress: totalCompactingKVs=500851692 less than currentCompactedKVs=568943946
> 2020-04-23 15:36:32,153 WARN org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress: totalCompactingKVs=500851692 less than currentCompactedKVs=570961155
> 2020-04-23 15:36:35,219 WARN org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress: totalCompactingKVs=500851692 less than currentCompactedKVs=573055384
> 2020-04-23 15:36:38,274 WARN org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress: totalCompactingKVs=500851692 less than currentCompactedKVs=575099844
> {code}
> This 'no state' is confusing... should be unknown region.
> {code}
> Why this message
> 2020-04-22 23:34:14,617 ERROR org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure: Error trying to merge [5aee12dd3e126b1fc4f5884be9712a5e, 1b4540d46c8ae2221759c72e328e3b28, 16f90525635f972bc984a498a17ceca4, f22469447941caf3d885b860d2f9ac8b, 971e35bff0c4edfd5416db246f499ac9, 441010c4577ee79c07a857e507567dc2, 94eaaee33dd2d22eebf40dec418e1f1c, d9a4fce4cc742a4d79cffb67a1b77510, b136369ec08d84d3761dcdde511c32e1, a6f6f29ce45d368fe49651723cacce10] in bobby_analytics (in state=MERGE_TABLE_REGIONS_PREPARE)
> org.apache.hadoop.hbase.UnknownRegionException: No state for [5aee12dd3e126b1fc4f5884be9712a5e]
> at org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.prepareMergeRegion(MergeTableRegionsProcedure.java:487)
> at org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:197)
> at org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:77)
> at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:194)
> at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:962)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1669)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1416)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:79)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1986)
> {code}
> Fix this log so can just copy paste it to shell... fix quotes:
> org.apache.hadoop.hbase.exceptions.MergeRegionException: Unable to merge non-adjacent or non-overlapping regions 9090c9927244d39d08dcf24335d2aa48, 04dc9c18f74937d2359e4ac2aaee9937 when force=false
--
This message was sent by Atlassian Jira
(v8.3.4#803005)