You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Michael Stack (Jira)" <ji...@apache.org> on 2020/04/23 20:55:00 UTC
[jira] [Created] (HBASE-24246) Miscellaneous hbck2 fixMeta bulk
merge fixes: better logging around merges/overlap-fixing, 'HBCK Report'
overlap listing, and configuration
Michael Stack created HBASE-24246:
-------------------------------------
Summary: Miscellaneous hbck2 fixMeta bulk merge fixes: better logging around merges/overlap-fixing, 'HBCK Report' overlap listing, and configuration
Key: HBASE-24246
URL: https://issues.apache.org/jira/browse/HBASE-24246
Project: HBase
Issue Type: Bug
Reporter: Michael Stack
Bit of a grab bag of small items.
Improve log messages when bulk merge runs (particularly out of 'hbck2 fixMeta').
Change config so we up the merge maximum since current low max is hit easily. When its hit, its merge of merge of merges w/ confusing output while happening. LOG when we hit max!
{code}
private static final String MAX_MERGE_COUNT_KEY = "hbase.master.metafixer.max.merge.count";
private static final int MAX_MERGE_COUNT_DEFAULT = 10;
{code}
Fix the 'HBCK Report' page so it only logs encoded region names because if region name is large, the UI becomes unreadable.
Fixup some logging messages. As is, can confuse operator.
E.g. explain why having a merge qualifier blocks a region being merged up
{code}
org.apache.hadoop.hbase.exceptions.MergeRegionException: Skip merging [e1ded5983cb00c1a1a45e17414c5914c, 2634c71a10cf4a216b7c1608b7d24e6a, e7d40bb7ab50cdeb20118043aedf2cdc, a9ea228464bd8a40b8908cd263e2f6be, 544d32880131443eebb2a01691994b0d, a82d462a6edca634e4f8ecff4e8bc55c, 03e30d277a1ab977f93c103cb917c7d7, d0d5fb364838b00942448b89b3d802ff, 00ba78a9be4b27cf076622f49c45a43e, 338efcfc48928815423d7a6601b9e477], because parent [e1ded5983cb00c1a1a45e17414c5914c] has a merge qualifier
{code}
Add note to orphans page that you shouldn't try to address orphans until all meta issues are fixed (overlaps, holes).
Add example on how to fix orphans with bulkload to the orphans section in hbck report. See HBASE-23928.
Why the angle brackets in here:
{code}
Why angle brackets here? Usually for lists.
2020-04-23 15:21:13,471 INFO org.apache.hadoop.hbase.master.HMaster: Client=hbase//10.103.230.91 merge regions [e1ded5983cb00c1a1a45e17414c5914c], [2634c71a10cf4a216b7c1608b7d24e6a], [e7d40bb7ab50cdeb20118043aedf2cdc], [a9ea228464bd8a40b8908cd263e2f6be], [544d32880131443eebb2a01691994b0d], [a82d462a6edca634e4f8ecff4e8bc55c], [03e30d277a1ab977f93c103cb917c7d7], [d0d5fb364838b00942448b89b3d802ff], [50cbee16764bb09a9887a07d3a01304f], [338efcfc48928815423d7a6601b9e477]
{code}
These logs make no sense... need region name at least
{code}
What is this? Missing region
Whats it doing?
2020-04-23 15:36:19,896 WARN org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress: totalCompactingKVs=500851692 less than currentCompactedKVs=562814154
2020-04-23 15:36:22,953 WARN org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress: totalCompactingKVs=500851692 less than currentCompactedKVs=564904980
2020-04-23 15:36:26,011 WARN org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress: totalCompactingKVs=500851692 less than currentCompactedKVs=566948397
2020-04-23 15:36:29,071 WARN org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress: totalCompactingKVs=500851692 less than currentCompactedKVs=568943946
2020-04-23 15:36:32,153 WARN org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress: totalCompactingKVs=500851692 less than currentCompactedKVs=570961155
2020-04-23 15:36:35,219 WARN org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress: totalCompactingKVs=500851692 less than currentCompactedKVs=573055384
2020-04-23 15:36:38,274 WARN org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress: totalCompactingKVs=500851692 less than currentCompactedKVs=575099844
{code}
This 'no state' is confusing... should be unknown region.
{code}
Why this message
2020-04-22 23:34:14,617 ERROR org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure: Error trying to merge [5aee12dd3e126b1fc4f5884be9712a5e, 1b4540d46c8ae2221759c72e328e3b28, 16f90525635f972bc984a498a17ceca4, f22469447941caf3d885b860d2f9ac8b, 971e35bff0c4edfd5416db246f499ac9, 441010c4577ee79c07a857e507567dc2, 94eaaee33dd2d22eebf40dec418e1f1c, d9a4fce4cc742a4d79cffb67a1b77510, b136369ec08d84d3761dcdde511c32e1, a6f6f29ce45d368fe49651723cacce10] in bobby_analytics (in state=MERGE_TABLE_REGIONS_PREPARE)
org.apache.hadoop.hbase.UnknownRegionException: No state for [5aee12dd3e126b1fc4f5884be9712a5e]
at org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.prepareMergeRegion(MergeTableRegionsProcedure.java:487)
at org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:197)
at org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:77)
at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:194)
at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:962)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1669)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1416)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:79)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1986)
{code}
Fix this log so can just copy paste it to shell... fix quotes:
org.apache.hadoop.hbase.exceptions.MergeRegionException: Unable to merge non-adjacent or non-overlapping regions 9090c9927244d39d08dcf24335d2aa48, 04dc9c18f74937d2359e4ac2aaee9937 when force=false
--
This message was sent by Atlassian Jira
(v8.3.4#803005)