You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2010/12/13 21:58:01 UTC
[jira] Created: (HBASE-3345) Master crash with NPE during table
disable
Master crash with NPE during table disable
------------------------------------------
Key: HBASE-3345
URL: https://issues.apache.org/jira/browse/HBASE-3345
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.0
Reporter: Todd Lipcon
Priority: Blocker
Running on a config that triggers lots of splits, I attempted to disable a table while it was getting a lot of load and injected failures. Got the following NPE in master, followed by an abort:
2010-12-13 12:52:27,323 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of region usertable,user1182862181,1292273503885.d223f1dc4d9003508f2db7566518b05d. (offlining)
2010-12-13 12:52:27,323 FATAL org.apache.hadoop.hbase.master.HMaster: Remote unexpected exception
java.lang.NullPointerException: Passed server is null
at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:581)
at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1085)
at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1032)
at org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler$1.run(DisableTableHandler.java:132)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (HBASE-3345) Master crash with NPE during table
disable
Posted by "Alex Newman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015651#comment-13015651 ]
Alex Newman commented on HBASE-3345:
------------------------------------
Most likely it wasn't that it was null but that the key didn't exist.
> Master crash with NPE during table disable
> ------------------------------------------
>
> Key: HBASE-3345
> URL: https://issues.apache.org/jira/browse/HBASE-3345
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Priority: Blocker
>
> Running on a config that triggers lots of splits, I attempted to disable a table while it was getting a lot of load and injected failures. Got the following NPE in master, followed by an abort:
> 2010-12-13 12:52:27,323 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of region usertable,user1182862181,1292273503885.d223f1dc4d9003508f2db7566518b05d. (offlining)
> 2010-12-13 12:52:27,323 FATAL org.apache.hadoop.hbase.master.HMaster: Remote unexpected exception
> java.lang.NullPointerException: Passed server is null
> at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:581)
> at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1085)
> at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1032)
> at org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler$1.run(DisableTableHandler.java:132)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-3345) Master crash with NPE during table
disable
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-3345.
--------------------------
Resolution: Won't Fix
Resolving. Think this is fixed. Will open new issue if see it again.
> Master crash with NPE during table disable
> ------------------------------------------
>
> Key: HBASE-3345
> URL: https://issues.apache.org/jira/browse/HBASE-3345
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Priority: Blocker
>
> Running on a config that triggers lots of splits, I attempted to disable a table while it was getting a lot of load and injected failures. Got the following NPE in master, followed by an abort:
> 2010-12-13 12:52:27,323 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of region usertable,user1182862181,1292273503885.d223f1dc4d9003508f2db7566518b05d. (offlining)
> 2010-12-13 12:52:27,323 FATAL org.apache.hadoop.hbase.master.HMaster: Remote unexpected exception
> java.lang.NullPointerException: Passed server is null
> at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:581)
> at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1085)
> at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1032)
> at org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler$1.run(DisableTableHandler.java:132)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3345) Master crash with NPE during table
disable
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13019703#comment-13019703 ]
stack commented on HBASE-3345:
------------------------------
We haven't seen this in a while. I think we've taken care of it and we should close this issue.
> Master crash with NPE during table disable
> ------------------------------------------
>
> Key: HBASE-3345
> URL: https://issues.apache.org/jira/browse/HBASE-3345
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Priority: Blocker
>
> Running on a config that triggers lots of splits, I attempted to disable a table while it was getting a lot of load and injected failures. Got the following NPE in master, followed by an abort:
> 2010-12-13 12:52:27,323 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of region usertable,user1182862181,1292273503885.d223f1dc4d9003508f2db7566518b05d. (offlining)
> 2010-12-13 12:52:27,323 FATAL org.apache.hadoop.hbase.master.HMaster: Remote unexpected exception
> java.lang.NullPointerException: Passed server is null
> at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:581)
> at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1085)
> at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1032)
> at org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler$1.run(DisableTableHandler.java:132)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HBASE-3345) Master crash with NPE during table
disable
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12971980#action_12971980 ]
stack commented on HBASE-3345:
------------------------------
I should have gotten context from you Todd when I was sitting beside you. We could make a change like this:
{code}
diff --git a/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java b/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
index ea064f2..1df65e7 100644
--- a/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
+++ b/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
@@ -1079,6 +1079,12 @@ public class AssignmentManager extends ZooKeeperListener {
HServerInfo server = null;
synchronized (this.regions) {
server = regions.get(region);
+ if (server == null) {
+ LOG.error("Can't unassign region " + region.getRegionNameAsString() +
+ " from a null server; removing from regions!");
+ regions.remove(region);
+ return;
+ }
}
try {
// TODO: We should consider making this look more like it does for the
{code}
... so we flag the problem earlier but just keep going. I wonder how server got to be null at all. Thats what I'd look in logs for.
> Master crash with NPE during table disable
> ------------------------------------------
>
> Key: HBASE-3345
> URL: https://issues.apache.org/jira/browse/HBASE-3345
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Priority: Blocker
>
> Running on a config that triggers lots of splits, I attempted to disable a table while it was getting a lot of load and injected failures. Got the following NPE in master, followed by an abort:
> 2010-12-13 12:52:27,323 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of region usertable,user1182862181,1292273503885.d223f1dc4d9003508f2db7566518b05d. (offlining)
> 2010-12-13 12:52:27,323 FATAL org.apache.hadoop.hbase.master.HMaster: Remote unexpected exception
> java.lang.NullPointerException: Passed server is null
> at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:581)
> at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1085)
> at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1032)
> at org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler$1.run(DisableTableHandler.java:132)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (HBASE-3345) Master crash with NPE during table
disable
Posted by "Alex Newman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081297#comment-13081297 ]
Alex Newman commented on HBASE-3345:
------------------------------------
agreed
> Master crash with NPE during table disable
> ------------------------------------------
>
> Key: HBASE-3345
> URL: https://issues.apache.org/jira/browse/HBASE-3345
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Priority: Blocker
>
> Running on a config that triggers lots of splits, I attempted to disable a table while it was getting a lot of load and injected failures. Got the following NPE in master, followed by an abort:
> 2010-12-13 12:52:27,323 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of region usertable,user1182862181,1292273503885.d223f1dc4d9003508f2db7566518b05d. (offlining)
> 2010-12-13 12:52:27,323 FATAL org.apache.hadoop.hbase.master.HMaster: Remote unexpected exception
> java.lang.NullPointerException: Passed server is null
> at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:581)
> at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1085)
> at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1032)
> at org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler$1.run(DisableTableHandler.java:132)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira