[jira] [Updated] (HBASE-11396) Invalid meta entries can lead to unstartable master

["Craig Condit (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-11396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Craig Condit updated HBASE-11396:
---------------------------------

    Attachment: HBASE-11396-v1.patch

Patch which catches the exception, logs an error, and returns null so that Master can continue startup.

> Invalid meta entries can lead to unstartable master
> ---------------------------------------------------
>
>                 Key: HBASE-11396
>                 URL: https://issues.apache.org/jira/browse/HBASE-11396
>             Project: HBase
>          Issue Type: Bug
>          Components: master
>    Affects Versions: 0.98.3
>            Reporter: Craig Condit
>         Attachments: HBASE-11396-v1.patch
>
>
> Recently I accidentally kill -9'd all regionservers in my cluster (don't ask.. not my finest moment).
> This let to some corruption of the meta table, causing the following exception to be output on the HBase Master during startup, and culminating with the Master aborting:
> {noformat}
> java.lang.IllegalArgumentException: Wrong length: 13, expected 8
> 	at org.apache.hadoop.hbase.util.Bytes.explainWrongLengthOrOffset(Bytes.java:600)
> 	at org.apache.hadoop.hbase.util.Bytes.toLong(Bytes.java:578)
> 	at org.apache.hadoop.hbase.HRegionInfo.getServerName(HRegionInfo.java:1059)
> 	at org.apache.hadoop.hbase.HRegionInfo.getHRegionInfoAndServerName(HRegionInfo.java:987)
> 	at org.apache.hadoop.hbase.master.AssignmentManager.rebuildUserRegions(AssignmentManager.java:2678)
> 	at org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:465)
> 	at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:910)
> 	at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:615)
> 	at java.lang.Thread.run(Thread.java:744)
> {noformat}
> No tools that I am aware of were able to clean this up. I added a short patch to catch and log this exception and return null from HRegionInfo.getServerName(). This allowed for startup of the cluster, and hbase hbck to repair the damage.
> Creating ticket to submit patch in case anyone else finds this useful.



--
This message was sent by Atlassian JIRA
(v6.2#6252)