You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2010/12/30 23:55:45 UTC
[jira] Created: (HBASE-3403) Region got undeployed
Region got undeployed
---------------------
Key: HBASE-3403
URL: https://issues.apache.org/jira/browse/HBASE-3403
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.0
Reporter: Todd Lipcon
Fix For: 0.90.0
ERROR: Region hdfs://haus01.sf.cloudera.com:11020/hbase-normal/usertable/2ad8df700eea55f70e02ea89178a65a2 on HDFS, but not listed in META or deployed on any region server.
ERROR: Found inconsistency in table usertable
Not sure how I got into this state, will look through logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3403) Region orphaned after failure during
split
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12979273#action_12979273 ]
Hudson commented on HBASE-3403:
-------------------------------
Integrated in HBase-TRUNK #1711 (See [https://hudson.apache.org/hudson/job/HBase-TRUNK/1711/])
HBASE-3403 Region orphaned after failure during split
> Region orphaned after failure during split
> ------------------------------------------
>
> Key: HBASE-3403
> URL: https://issues.apache.org/jira/browse/HBASE-3403
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Assignee: stack
> Priority: Blocker
> Fix For: 0.90.0
>
> Attachments: 3403-v2.txt, 3403.txt, broken-split.txt, hbck-fix-missing-in-meta.txt, master-logs.txt.gz
>
>
> ERROR: Region hdfs://haus01.sf.cloudera.com:11020/hbase-normal/usertable/2ad8df700eea55f70e02ea89178a65a2 on HDFS, but not listed in META or deployed on any region server.
> ERROR: Found inconsistency in table usertable
> Not sure how I got into this state, will look through logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3403) Region orphaned after failure during
split
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3403:
-------------------------
Attachment: 3403.txt
Patch that fixes issue where we are clearing server from .META. in parent of split so if crash, we fail to do fixup if daughters didn't get added properly.
Adds ugly test that bends over backwards to recreate the scenario (we have some work to do to expose our innards for mocking).
> Region orphaned after failure during split
> ------------------------------------------
>
> Key: HBASE-3403
> URL: https://issues.apache.org/jira/browse/HBASE-3403
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Priority: Blocker
> Fix For: 0.90.0
>
> Attachments: 3403.txt, broken-split.txt, hbck-fix-missing-in-meta.txt, master-logs.txt.gz
>
>
> ERROR: Region hdfs://haus01.sf.cloudera.com:11020/hbase-normal/usertable/2ad8df700eea55f70e02ea89178a65a2 on HDFS, but not listed in META or deployed on any region server.
> ERROR: Found inconsistency in table usertable
> Not sure how I got into this state, will look through logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3403) Region orphaned after failure during
split
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12976185#action_12976185 ]
stack commented on HBASE-3403:
------------------------------
There is fixup for this condition in the server shutdown recovery code. See the fixupDaughters methods in the ServerShutdown handler. I noticed recently that it doesn't seem to be working properly (I noted it in issue comment but didn't open explicit issue to cover this case since couldn't repro). I added extra logging to see if could figure why the fixup code wasn't cutting in. Mind posting master log? Good on you Todd.
> Region orphaned after failure during split
> ------------------------------------------
>
> Key: HBASE-3403
> URL: https://issues.apache.org/jira/browse/HBASE-3403
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Fix For: 0.90.0
>
> Attachments: broken-split.txt, hbck-fix-missing-in-meta.txt
>
>
> ERROR: Region hdfs://haus01.sf.cloudera.com:11020/hbase-normal/usertable/2ad8df700eea55f70e02ea89178a65a2 on HDFS, but not listed in META or deployed on any region server.
> ERROR: Found inconsistency in table usertable
> Not sure how I got into this state, will look through logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3403) Region orphaned after failure during
split
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978035#action_12978035 ]
stack commented on HBASE-3403:
------------------------------
bq. + cluster.getMaster().catalogJanitorSwitch(false);
I added above switch to avoid the unlikely but *perhaps* possible case of split, compaction in each daughter, and run of catalogjanitor happens before we get our edit of .META. in. Just trying to do all I can to avoid a debug of failed tests up on hudson.
NP on changing name of method. Will do.
bq. Does this change introduce a new bug?
Yes. That could happen. Unlikely, but perhaps. Let me spin a new patch.
> Region orphaned after failure during split
> ------------------------------------------
>
> Key: HBASE-3403
> URL: https://issues.apache.org/jira/browse/HBASE-3403
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Priority: Blocker
> Fix For: 0.90.0
>
> Attachments: 3403.txt, broken-split.txt, hbck-fix-missing-in-meta.txt, master-logs.txt.gz
>
>
> ERROR: Region hdfs://haus01.sf.cloudera.com:11020/hbase-normal/usertable/2ad8df700eea55f70e02ea89178a65a2 on HDFS, but not listed in META or deployed on any region server.
> ERROR: Found inconsistency in table usertable
> Not sure how I got into this state, will look through logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3403) Region orphaned after failure during
split
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HBASE-3403:
-------------------------------
Summary: Region orphaned after failure during split (was: Region got undeployed)
(fwiw hbck -fix cannot fix this class of issues)
> Region orphaned after failure during split
> ------------------------------------------
>
> Key: HBASE-3403
> URL: https://issues.apache.org/jira/browse/HBASE-3403
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Fix For: 0.90.0
>
> Attachments: broken-split.txt
>
>
> ERROR: Region hdfs://haus01.sf.cloudera.com:11020/hbase-normal/usertable/2ad8df700eea55f70e02ea89178a65a2 on HDFS, but not listed in META or deployed on any region server.
> ERROR: Found inconsistency in table usertable
> Not sure how I got into this state, will look through logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3403) Region orphaned after failure during
split
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977006#action_12977006 ]
stack commented on HBASE-3403:
------------------------------
The log does not have important transition points for the above cited problematic region nor for the few that are stuck PENDING_CLOSE on end of the log. Let me try reproduce. Thanks Todd.
> Region orphaned after failure during split
> ------------------------------------------
>
> Key: HBASE-3403
> URL: https://issues.apache.org/jira/browse/HBASE-3403
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Fix For: 0.90.0
>
> Attachments: broken-split.txt, hbck-fix-missing-in-meta.txt, master-logs.txt.gz
>
>
> ERROR: Region hdfs://haus01.sf.cloudera.com:11020/hbase-normal/usertable/2ad8df700eea55f70e02ea89178a65a2 on HDFS, but not listed in META or deployed on any region server.
> ERROR: Found inconsistency in table usertable
> Not sure how I got into this state, will look through logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3403) Region orphaned after failure during
split
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HBASE-3403:
-------------------------------
Priority: Blocker (was: Major)
> Region orphaned after failure during split
> ------------------------------------------
>
> Key: HBASE-3403
> URL: https://issues.apache.org/jira/browse/HBASE-3403
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Priority: Blocker
> Fix For: 0.90.0
>
> Attachments: broken-split.txt, hbck-fix-missing-in-meta.txt, master-logs.txt.gz
>
>
> ERROR: Region hdfs://haus01.sf.cloudera.com:11020/hbase-normal/usertable/2ad8df700eea55f70e02ea89178a65a2 on HDFS, but not listed in META or deployed on any region server.
> ERROR: Found inconsistency in table usertable
> Not sure how I got into this state, will look through logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3403) Region orphaned after failure during
split
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HBASE-3403:
-------------------------------
Attachment: hbck-fix-missing-in-meta.txt
Here's an hbck fix that will re-add a region to META. It should probably be improved to actually double-check that META has a hole for the region where it's adding. Otherwise it could potentially do more harm than good!
> Region orphaned after failure during split
> ------------------------------------------
>
> Key: HBASE-3403
> URL: https://issues.apache.org/jira/browse/HBASE-3403
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Fix For: 0.90.0
>
> Attachments: broken-split.txt, hbck-fix-missing-in-meta.txt
>
>
> ERROR: Region hdfs://haus01.sf.cloudera.com:11020/hbase-normal/usertable/2ad8df700eea55f70e02ea89178a65a2 on HDFS, but not listed in META or deployed on any region server.
> ERROR: Found inconsistency in table usertable
> Not sure how I got into this state, will look through logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3403) Region got undeployed
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12976162#action_12976162 ]
Todd Lipcon commented on HBASE-3403:
------------------------------------
Worth noting that I kill -9ed all the servers at just about 11:56:46,879.
> Region got undeployed
> ---------------------
>
> Key: HBASE-3403
> URL: https://issues.apache.org/jira/browse/HBASE-3403
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Fix For: 0.90.0
>
> Attachments: broken-split.txt
>
>
> ERROR: Region hdfs://haus01.sf.cloudera.com:11020/hbase-normal/usertable/2ad8df700eea55f70e02ea89178a65a2 on HDFS, but not listed in META or deployed on any region server.
> ERROR: Found inconsistency in table usertable
> Not sure how I got into this state, will look through logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3403) Region got undeployed
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12976168#action_12976168 ]
Todd Lipcon commented on HBASE-3403:
------------------------------------
I think the RS crashed somewhere between offlining the parent and opening one of the new daughters, where this comment is:
{code}
// The is the point of no return. We are committed to the split now. We
// have still the daughter regions to open but meta has been changed.
// If we fail from here on out, we can not rollback so, we'll just abort.
// The meta has been changed though so there will need to be a fixup run
// during processing of the crashed server by master (TODO: Verify this in place).
{code}
When the master processed the RS failure, the daughter region wasn't yet in META, so it didn't count them as unassigned during processing the server shutdown.
Perhaps MetaReader.getServerUserRegions() needs to take into account offline parents. If it finds an offline parent, it should add the two daughters to a list of potentially mid-split regions. Then it can issue a get() for each of those regions, and if it doesn't find a row, it may need to be counted as unassigned... seems a bit messy :)
> Region got undeployed
> ---------------------
>
> Key: HBASE-3403
> URL: https://issues.apache.org/jira/browse/HBASE-3403
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Fix For: 0.90.0
>
> Attachments: broken-split.txt
>
>
> ERROR: Region hdfs://haus01.sf.cloudera.com:11020/hbase-normal/usertable/2ad8df700eea55f70e02ea89178a65a2 on HDFS, but not listed in META or deployed on any region server.
> ERROR: Found inconsistency in table usertable
> Not sure how I got into this state, will look through logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3403) Region orphaned after failure during
split
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3403:
-------------------------
Attachment: 3403-v2.txt
Made Todd suggested changes as well as handling of the possible scenario he describes. The fixup code was copied from 0.20. In 0.20, it was not possible to get into the state Todd postulates above but in 0.90, when fixup is done in shutdown handler and not in the 'catalogJanitor', it could happen. I added test that manufactures the postulated condition and we seem to be doing right thing.
> Region orphaned after failure during split
> ------------------------------------------
>
> Key: HBASE-3403
> URL: https://issues.apache.org/jira/browse/HBASE-3403
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Priority: Blocker
> Fix For: 0.90.0
>
> Attachments: 3403-v2.txt, 3403.txt, broken-split.txt, hbck-fix-missing-in-meta.txt, master-logs.txt.gz
>
>
> ERROR: Region hdfs://haus01.sf.cloudera.com:11020/hbase-normal/usertable/2ad8df700eea55f70e02ea89178a65a2 on HDFS, but not listed in META or deployed on any region server.
> ERROR: Found inconsistency in table usertable
> Not sure how I got into this state, will look through logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3403) Region orphaned after failure during
split
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977989#action_12977989 ]
Todd Lipcon commented on HBASE-3403:
------------------------------------
+ // Turn off the meta scanner so it don't remove parent on us.
+ cluster.getMaster().catalogJanitorSwitch(false);
Is this a potential issue? ie the catalog janitor shouldn't remove the parent if the daughters aren't ready yet, should it? Or is this just for the simulation done in the test?
- naming-wise, can we use "setCatalogJanitorEnabled(false)" just like our other accessors? Also .onOff() -> setEnabled()?
- Does this change introduce a new bug? Consider the following scenario:
1. Region A exists
2. Region A splits to B and C, A is marked offline
3. B and C both open just fine
4. B splits to E and F, successfully
5. server crashes
When we process the server shutdown, we end up re-creating B.
> Region orphaned after failure during split
> ------------------------------------------
>
> Key: HBASE-3403
> URL: https://issues.apache.org/jira/browse/HBASE-3403
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Priority: Blocker
> Fix For: 0.90.0
>
> Attachments: 3403.txt, broken-split.txt, hbck-fix-missing-in-meta.txt, master-logs.txt.gz
>
>
> ERROR: Region hdfs://haus01.sf.cloudera.com:11020/hbase-normal/usertable/2ad8df700eea55f70e02ea89178a65a2 on HDFS, but not listed in META or deployed on any region server.
> ERROR: Found inconsistency in table usertable
> Not sure how I got into this state, will look through logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3403) Region got undeployed
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HBASE-3403:
-------------------------------
Attachment: broken-split.txt
It seems this region was involved in a split (starts at 2010-12-30 11:55:15,035). Here are all the logs for the parent (1acb10b8287adbd1a2ace40331366a8c) and the two daughters
> Region got undeployed
> ---------------------
>
> Key: HBASE-3403
> URL: https://issues.apache.org/jira/browse/HBASE-3403
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Fix For: 0.90.0
>
> Attachments: broken-split.txt
>
>
> ERROR: Region hdfs://haus01.sf.cloudera.com:11020/hbase-normal/usertable/2ad8df700eea55f70e02ea89178a65a2 on HDFS, but not listed in META or deployed on any region server.
> ERROR: Found inconsistency in table usertable
> Not sure how I got into this state, will look through logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3403) Region orphaned after failure during
split
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HBASE-3403:
-------------------------------
Attachment: master-logs.txt.gz
Here are some logs from the master
> Region orphaned after failure during split
> ------------------------------------------
>
> Key: HBASE-3403
> URL: https://issues.apache.org/jira/browse/HBASE-3403
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Fix For: 0.90.0
>
> Attachments: broken-split.txt, hbck-fix-missing-in-meta.txt, master-logs.txt.gz
>
>
> ERROR: Region hdfs://haus01.sf.cloudera.com:11020/hbase-normal/usertable/2ad8df700eea55f70e02ea89178a65a2 on HDFS, but not listed in META or deployed on any region server.
> ERROR: Found inconsistency in table usertable
> Not sure how I got into this state, will look through logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3403) Region orphaned after failure during
split
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12979821#action_12979821 ]
stack commented on HBASE-3403:
------------------------------
Duh, forgot to add Test to 0.90. Just did. Thanks Ted Yu for noticing.
> Region orphaned after failure during split
> ------------------------------------------
>
> Key: HBASE-3403
> URL: https://issues.apache.org/jira/browse/HBASE-3403
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Assignee: stack
> Priority: Blocker
> Fix For: 0.90.0
>
> Attachments: 3403-v2.txt, 3403.txt, broken-split.txt, hbck-fix-missing-in-meta.txt, master-logs.txt.gz
>
>
> ERROR: Region hdfs://haus01.sf.cloudera.com:11020/hbase-normal/usertable/2ad8df700eea55f70e02ea89178a65a2 on HDFS, but not listed in META or deployed on any region server.
> ERROR: Found inconsistency in table usertable
> Not sure how I got into this state, will look through logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3403) Region orphaned after failure during
split
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12979266#action_12979266 ]
stack commented on HBASE-3403:
------------------------------
Really committed to TRUNK (I'd only committed on branch as noticed by Ted Yu. Thanks Hawkeye Ted).
> Region orphaned after failure during split
> ------------------------------------------
>
> Key: HBASE-3403
> URL: https://issues.apache.org/jira/browse/HBASE-3403
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Assignee: stack
> Priority: Blocker
> Fix For: 0.90.0
>
> Attachments: 3403-v2.txt, 3403.txt, broken-split.txt, hbck-fix-missing-in-meta.txt, master-logs.txt.gz
>
>
> ERROR: Region hdfs://haus01.sf.cloudera.com:11020/hbase-normal/usertable/2ad8df700eea55f70e02ea89178a65a2 on HDFS, but not listed in META or deployed on any region server.
> ERROR: Found inconsistency in table usertable
> Not sure how I got into this state, will look through logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-3403) Region orphaned after failure during
split
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-3403.
--------------------------
Resolution: Fixed
Assignee: stack
Hadoop Flags: [Reviewed]
Committed branch and trunk after making the new Todd suggested changes. Thanks for review Todd.
> Region orphaned after failure during split
> ------------------------------------------
>
> Key: HBASE-3403
> URL: https://issues.apache.org/jira/browse/HBASE-3403
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Assignee: stack
> Priority: Blocker
> Fix For: 0.90.0
>
> Attachments: 3403-v2.txt, 3403.txt, broken-split.txt, hbck-fix-missing-in-meta.txt, master-logs.txt.gz
>
>
> ERROR: Region hdfs://haus01.sf.cloudera.com:11020/hbase-normal/usertable/2ad8df700eea55f70e02ea89178a65a2 on HDFS, but not listed in META or deployed on any region server.
> ERROR: Found inconsistency in table usertable
> Not sure how I got into this state, will look through logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3403) Region orphaned after failure during
split
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977038#action_12977038 ]
Todd Lipcon commented on HBASE-3403:
------------------------------------
Stack and I looked at this. Here's the issue:
- when we offline the parent in MetaEditor, we clear out SERVER_QUALIFIER and SERVER_STARTCODE
- so, when we process the server shutdown in ServerShutdownHandler, it never makes it into the list of regions that were on that server.
- hence, we never call processDeadRegion, and it gets stuck.
Stack is going to try to write a unit test.
> Region orphaned after failure during split
> ------------------------------------------
>
> Key: HBASE-3403
> URL: https://issues.apache.org/jira/browse/HBASE-3403
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Fix For: 0.90.0
>
> Attachments: broken-split.txt, hbck-fix-missing-in-meta.txt, master-logs.txt.gz
>
>
> ERROR: Region hdfs://haus01.sf.cloudera.com:11020/hbase-normal/usertable/2ad8df700eea55f70e02ea89178a65a2 on HDFS, but not listed in META or deployed on any region server.
> ERROR: Found inconsistency in table usertable
> Not sure how I got into this state, will look through logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3403) Region orphaned after failure during
split
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978915#action_12978915 ]
Todd Lipcon commented on HBASE-3403:
------------------------------------
Looks pretty good! A few nits:
- CatalogJanitor.setEnabled() - why does it return its argument? that seems kind of silly... Same with HMaster.setCatalogJanitorEnabled.
- typo: "used testing" -> "Used for testing"
- typo: "start the scan at the what"
- rename: isDaughter -> foundDaughter, IsDaughterVisitor -> FindDaughterVisitor might make more sense?
> Region orphaned after failure during split
> ------------------------------------------
>
> Key: HBASE-3403
> URL: https://issues.apache.org/jira/browse/HBASE-3403
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Priority: Blocker
> Fix For: 0.90.0
>
> Attachments: 3403-v2.txt, 3403.txt, broken-split.txt, hbck-fix-missing-in-meta.txt, master-logs.txt.gz
>
>
> ERROR: Region hdfs://haus01.sf.cloudera.com:11020/hbase-normal/usertable/2ad8df700eea55f70e02ea89178a65a2 on HDFS, but not listed in META or deployed on any region server.
> ERROR: Found inconsistency in table usertable
> Not sure how I got into this state, will look through logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.