You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2008/03/11 21:50:49 UTC
[jira] Created: (HBASE-503) cluster won't shut down
cluster won't shut down
-----------------------
Key: HBASE-503
URL: https://issues.apache.org/jira/browse/HBASE-503
Project: Hadoop HBase
Issue Type: Bug
Affects Versions: 0.1.0, 0.2.0, 0.16.0
Reporter: stack
Master is stuck trying to shutdown. It gets confused if its not running the shutdown. Scenario is cluster is being monitored by a watcher process. When a server goes down, its restarted. In this environment, all hbase was updated then each server was restarted. The regionservers bounced fine but the master won't go down. Its stuck servicing reports of newly started regionservers to whom it sends a shutdown.... but cluster is of such a size that the master hasn't gone down by the time the regionserver starts again. Here is how the master log looks for one server:
{code}
2008-03-11 20:47:08,198 INFO org.apache.hadoop.hbase.HMaster: Cancelling lease for XX.XX.XX.122:60020
2008-03-11 20:47:08,198 INFO org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- lease cancelled
2008-03-11 20:47:08,398 DEBUG org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- cancelling lease
2008-03-11 20:47:16,421 INFO org.apache.hadoop.hbase.HMaster: received start message from: XX.XX.XX.122:60020
2008-03-11 20:47:20,163 DEBUG org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- cancelling lease
2008-03-11 20:47:20,163 INFO org.apache.hadoop.hbase.HMaster: Cancelling lease for XX.XX.XX.122:60020
2008-03-11 20:47:20,163 INFO org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- lease cancelled
2008-03-11 20:47:20,393 DEBUG org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- cancelling lease
2008-03-11 20:47:28,374 INFO org.apache.hadoop.hbase.HMaster: received start message from: XX.XX.XX.122:600
202008-03-11 20:47:32,095 DEBUG org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- cancelling lease
2008-03-11 20:47:32,095 INFO org.apache.hadoop.hbase.HMaster: Cancelling lease for XX.XX.XX.122:60020
2008-03-11 20:47:32,095 INFO org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- lease cancelled
2008-03-11 20:47:32,274 DEBUG org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- cancelling lease
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-503) cluster won't shut down
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-503:
------------------------
Priority: Minor (was: Major)
Marking minor. The babysitting process was root cause. We should be able to deal though.
> cluster won't shut down
> -----------------------
>
> Key: HBASE-503
> URL: https://issues.apache.org/jira/browse/HBASE-503
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.16.0, 0.1.0, 0.2.0
> Reporter: stack
> Priority: Minor
>
> Master is stuck trying to shutdown. It gets confused if its not running the shutdown. Scenario is cluster is being monitored by a watcher process. When a server goes down, its restarted. In this environment, all hbase was updated then each server was restarted. The regionservers bounced fine but the master won't go down. Its stuck servicing reports of newly started regionservers to whom it sends a shutdown.... but cluster is of such a size that the master hasn't gone down by the time the regionserver starts again. Here is how the master log looks for one server:
> {code}
> 2008-03-11 20:47:08,198 INFO org.apache.hadoop.hbase.HMaster: Cancelling lease for XX.XX.XX.122:60020
> 2008-03-11 20:47:08,198 INFO org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- lease cancelled
> 2008-03-11 20:47:08,398 DEBUG org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- cancelling lease
> 2008-03-11 20:47:16,421 INFO org.apache.hadoop.hbase.HMaster: received start message from: XX.XX.XX.122:60020
> 2008-03-11 20:47:20,163 DEBUG org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- cancelling lease
> 2008-03-11 20:47:20,163 INFO org.apache.hadoop.hbase.HMaster: Cancelling lease for XX.XX.XX.122:60020
> 2008-03-11 20:47:20,163 INFO org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- lease cancelled
> 2008-03-11 20:47:20,393 DEBUG org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- cancelling lease
> 2008-03-11 20:47:28,374 INFO org.apache.hadoop.hbase.HMaster: received start message from: XX.XX.XX.122:600
> 202008-03-11 20:47:32,095 DEBUG org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- cancelling lease
> 2008-03-11 20:47:32,095 INFO org.apache.hadoop.hbase.HMaster: Cancelling lease for XX.XX.XX.122:60020
> 2008-03-11 20:47:32,095 INFO org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- lease cancelled
> 2008-03-11 20:47:32,274 DEBUG org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- cancelling lease
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-503) cluster won't shut down
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-503.
-------------------------
Resolution: Won't Fix
Issue is too old and original description was about a process watched by errant babysitter.
> cluster won't shut down
> -----------------------
>
> Key: HBASE-503
> URL: https://issues.apache.org/jira/browse/HBASE-503
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.16.0, 0.1.0, 0.2.0
> Reporter: stack
> Priority: Minor
>
> Master is stuck trying to shutdown. It gets confused if its not running the shutdown. Scenario is cluster is being monitored by a watcher process. When a server goes down, its restarted. In this environment, all hbase was updated then each server was restarted. The regionservers bounced fine but the master won't go down. Its stuck servicing reports of newly started regionservers to whom it sends a shutdown.... but cluster is of such a size that the master hasn't gone down by the time the regionserver starts again. Here is how the master log looks for one server:
> {code}
> 2008-03-11 20:47:08,198 INFO org.apache.hadoop.hbase.HMaster: Cancelling lease for XX.XX.XX.122:60020
> 2008-03-11 20:47:08,198 INFO org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- lease cancelled
> 2008-03-11 20:47:08,398 DEBUG org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- cancelling lease
> 2008-03-11 20:47:16,421 INFO org.apache.hadoop.hbase.HMaster: received start message from: XX.XX.XX.122:60020
> 2008-03-11 20:47:20,163 DEBUG org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- cancelling lease
> 2008-03-11 20:47:20,163 INFO org.apache.hadoop.hbase.HMaster: Cancelling lease for XX.XX.XX.122:60020
> 2008-03-11 20:47:20,163 INFO org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- lease cancelled
> 2008-03-11 20:47:20,393 DEBUG org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- cancelling lease
> 2008-03-11 20:47:28,374 INFO org.apache.hadoop.hbase.HMaster: received start message from: XX.XX.XX.122:600
> 202008-03-11 20:47:32,095 DEBUG org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- cancelling lease
> 2008-03-11 20:47:32,095 INFO org.apache.hadoop.hbase.HMaster: Cancelling lease for XX.XX.XX.122:60020
> 2008-03-11 20:47:32,095 INFO org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- lease cancelled
> 2008-03-11 20:47:32,274 DEBUG org.apache.hadoop.hbase.HMaster: Region server XX.XX.XX.122:60020: MSG_REPORT_EXITING -- cancelling lease
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.