You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2011/01/05 18:52:45 UTC
[jira] Created: (HBASE-3420) Handling a big rebalance, we can queue
multiple instances of a Close event; messes up state
Handling a big rebalance, we can queue multiple instances of a Close event; messes up state
-------------------------------------------------------------------------------------------
Key: HBASE-3420
URL: https://issues.apache.org/jira/browse/HBASE-3420
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.0
Reporter: stack
Fix For: 0.90.1
This is pretty ugly. In short, on a heavily loaded cluster, we are queuing multiple instances of region close. They all try to run confusing state.
Long version:
I have a messy cluster. Its 16k regions on 8 servers. One node has 5k or so regions on it. Heaps are 1G all around. My master had OOME'd. Not sure why but not too worried about it for now. So, new master comes up and is trying to rebalance the cluster:
{code}
2011-01-05 00:48:07,385 INFO org.apache.hadoop.hbase.master.LoadBalancer: Calculated a load balance in 14ms. Moving 3666 regions off of 6 overloaded servers onto 3 less loaded servers
{code}
The balancer ends up sending many closes to a single overloaded server are taking so long, the close times out in RIT. We then do this:
{code}
case CLOSED:
LOG.info("Region has been CLOSED for too long, " +
"retriggering ClosedRegionHandler");
AssignmentManager.this.executorService.submit(
new ClosedRegionHandler(master, AssignmentManager.this,
regionState.getRegion()));
break;
{code}
We queue a new close (Should we?).
We time out a few more times (9 times) and each time we queue a new close.
Eventually the close succeeds, the region gets assigned a new location.
Then the next close pops off the eventhandler queue.
Here is the telltale signature of stuff gone amiss:
{code}
2011-01-05 00:52:19,379 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. state=OPEN, ts=1294188709030
{code}
Notice how state is OPEN when we are forcing offline (It was actually just successfully opened). We end up assigning same server because plan was still around:
{code}
2011-01-05 00:52:20,705 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Attempted open of TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. but already online on this server
{code}
But later when plan is cleared, we assign new server and we have dbl-assignment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3420) Handling a big rebalance, we can
queue multiple instances of a Close event; messes up state
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977968#action_12977968 ]
stack commented on HBASE-3420:
------------------------------
Ok... with this patch in place, master was able to join the cluster w/o aborting and live through the rebalance (all regions cleared from RIT). I'm going to commit.
> Handling a big rebalance, we can queue multiple instances of a Close event; messes up state
> -------------------------------------------------------------------------------------------
>
> Key: HBASE-3420
> URL: https://issues.apache.org/jira/browse/HBASE-3420
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: stack
> Fix For: 0.90.1
>
> Attachments: 3420.txt
>
>
> This is pretty ugly. In short, on a heavily loaded cluster, we are queuing multiple instances of region close. They all try to run confusing state.
> Long version:
> I have a messy cluster. Its 16k regions on 8 servers. One node has 5k or so regions on it. Heaps are 1G all around. My master had OOME'd. Not sure why but not too worried about it for now. So, new master comes up and is trying to rebalance the cluster:
> {code}
> 2011-01-05 00:48:07,385 INFO org.apache.hadoop.hbase.master.LoadBalancer: Calculated a load balance in 14ms. Moving 3666 regions off of 6 overloaded servers onto 3 less loaded servers
> {code}
> The balancer ends up sending many closes to a single overloaded server are taking so long, the close times out in RIT. We then do this:
> {code}
> case CLOSED:
> LOG.info("Region has been CLOSED for too long, " +
> "retriggering ClosedRegionHandler");
> AssignmentManager.this.executorService.submit(
> new ClosedRegionHandler(master, AssignmentManager.this,
> regionState.getRegion()));
> break;
> {code}
> We queue a new close (Should we?).
> We time out a few more times (9 times) and each time we queue a new close.
> Eventually the close succeeds, the region gets assigned a new location.
> Then the next close pops off the eventhandler queue.
> Here is the telltale signature of stuff gone amiss:
> {code}
> 2011-01-05 00:52:19,379 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. state=OPEN, ts=1294188709030
> {code}
> Notice how state is OPEN when we are forcing offline (It was actually just successfully opened). We end up assigning same server because plan was still around:
> {code}
> 2011-01-05 00:52:20,705 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Attempted open of TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. but already online on this server
> {code}
> But later when plan is cleared, we assign new server and we have dbl-assignment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3420) Handling a big rebalance, we can queue
multiple instances of a Close event; messes up state
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3420:
-------------------------
Attachment: 3420.txt
This should address most egregious issue turned up by these logs. Other things to add are maximum regions to assign per balance. We should add that too. Will make a new issue for that once this goes in.
> Handling a big rebalance, we can queue multiple instances of a Close event; messes up state
> -------------------------------------------------------------------------------------------
>
> Key: HBASE-3420
> URL: https://issues.apache.org/jira/browse/HBASE-3420
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: stack
> Fix For: 0.90.1
>
> Attachments: 3420.txt
>
>
> This is pretty ugly. In short, on a heavily loaded cluster, we are queuing multiple instances of region close. They all try to run confusing state.
> Long version:
> I have a messy cluster. Its 16k regions on 8 servers. One node has 5k or so regions on it. Heaps are 1G all around. My master had OOME'd. Not sure why but not too worried about it for now. So, new master comes up and is trying to rebalance the cluster:
> {code}
> 2011-01-05 00:48:07,385 INFO org.apache.hadoop.hbase.master.LoadBalancer: Calculated a load balance in 14ms. Moving 3666 regions off of 6 overloaded servers onto 3 less loaded servers
> {code}
> The balancer ends up sending many closes to a single overloaded server are taking so long, the close times out in RIT. We then do this:
> {code}
> case CLOSED:
> LOG.info("Region has been CLOSED for too long, " +
> "retriggering ClosedRegionHandler");
> AssignmentManager.this.executorService.submit(
> new ClosedRegionHandler(master, AssignmentManager.this,
> regionState.getRegion()));
> break;
> {code}
> We queue a new close (Should we?).
> We time out a few more times (9 times) and each time we queue a new close.
> Eventually the close succeeds, the region gets assigned a new location.
> Then the next close pops off the eventhandler queue.
> Here is the telltale signature of stuff gone amiss:
> {code}
> 2011-01-05 00:52:19,379 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. state=OPEN, ts=1294188709030
> {code}
> Notice how state is OPEN when we are forcing offline (It was actually just successfully opened). We end up assigning same server because plan was still around:
> {code}
> 2011-01-05 00:52:20,705 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Attempted open of TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. but already online on this server
> {code}
> But later when plan is cleared, we assign new server and we have dbl-assignment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3420) Handling a big rebalance, we can
queue multiple instances of a Close event; messes up state
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977884#action_12977884 ]
Jonathan Gray commented on HBASE-3420:
--------------------------------------
Is this the timeout of a CLOSE or an OPEN? If the master state is CLOSED, that means the region is completely offline and so the next thing is to send an OPEN RPC.
As I see it, the acknowledgment of an OPEN RPC is the transitioning of the ZK node to OPENING. That is not happening within the timeout so it makes sense that this times out. (of course this is not great behavior but is "correct" as it stands now).
BUT we should deal properly with the sending of multiple OPEN RPCs to different RSs.
Why the force to offline? That seems to be at the heart of this problem. If we do a ClosedRegionHandler and the region is shown as OPEN then we should not modify the region state.
Perhaps we should not be doing the ClosedRegionHandler on the timeout (which is expected to be called after a ZK node is transitioned to CLOSED). Instead, we should just retry an assignment but w/o forcing the ZK node to a certain state and ensuring that the region is not already assigned.
> Handling a big rebalance, we can queue multiple instances of a Close event; messes up state
> -------------------------------------------------------------------------------------------
>
> Key: HBASE-3420
> URL: https://issues.apache.org/jira/browse/HBASE-3420
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: stack
> Fix For: 0.90.1
>
>
> This is pretty ugly. In short, on a heavily loaded cluster, we are queuing multiple instances of region close. They all try to run confusing state.
> Long version:
> I have a messy cluster. Its 16k regions on 8 servers. One node has 5k or so regions on it. Heaps are 1G all around. My master had OOME'd. Not sure why but not too worried about it for now. So, new master comes up and is trying to rebalance the cluster:
> {code}
> 2011-01-05 00:48:07,385 INFO org.apache.hadoop.hbase.master.LoadBalancer: Calculated a load balance in 14ms. Moving 3666 regions off of 6 overloaded servers onto 3 less loaded servers
> {code}
> The balancer ends up sending many closes to a single overloaded server are taking so long, the close times out in RIT. We then do this:
> {code}
> case CLOSED:
> LOG.info("Region has been CLOSED for too long, " +
> "retriggering ClosedRegionHandler");
> AssignmentManager.this.executorService.submit(
> new ClosedRegionHandler(master, AssignmentManager.this,
> regionState.getRegion()));
> break;
> {code}
> We queue a new close (Should we?).
> We time out a few more times (9 times) and each time we queue a new close.
> Eventually the close succeeds, the region gets assigned a new location.
> Then the next close pops off the eventhandler queue.
> Here is the telltale signature of stuff gone amiss:
> {code}
> 2011-01-05 00:52:19,379 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. state=OPEN, ts=1294188709030
> {code}
> Notice how state is OPEN when we are forcing offline (It was actually just successfully opened). We end up assigning same server because plan was still around:
> {code}
> 2011-01-05 00:52:20,705 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Attempted open of TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. but already online on this server
> {code}
> But later when plan is cleared, we assign new server and we have dbl-assignment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3420) Handling a big rebalance, we can
queue multiple instances of a Close event; messes up state
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977909#action_12977909 ]
Ted Yu commented on HBASE-3420:
-------------------------------
hbase.regions.percheckin is no longer in 0.90
Can we reuse it ?
> Handling a big rebalance, we can queue multiple instances of a Close event; messes up state
> -------------------------------------------------------------------------------------------
>
> Key: HBASE-3420
> URL: https://issues.apache.org/jira/browse/HBASE-3420
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: stack
> Fix For: 0.90.1
>
> Attachments: 3420.txt
>
>
> This is pretty ugly. In short, on a heavily loaded cluster, we are queuing multiple instances of region close. They all try to run confusing state.
> Long version:
> I have a messy cluster. Its 16k regions on 8 servers. One node has 5k or so regions on it. Heaps are 1G all around. My master had OOME'd. Not sure why but not too worried about it for now. So, new master comes up and is trying to rebalance the cluster:
> {code}
> 2011-01-05 00:48:07,385 INFO org.apache.hadoop.hbase.master.LoadBalancer: Calculated a load balance in 14ms. Moving 3666 regions off of 6 overloaded servers onto 3 less loaded servers
> {code}
> The balancer ends up sending many closes to a single overloaded server are taking so long, the close times out in RIT. We then do this:
> {code}
> case CLOSED:
> LOG.info("Region has been CLOSED for too long, " +
> "retriggering ClosedRegionHandler");
> AssignmentManager.this.executorService.submit(
> new ClosedRegionHandler(master, AssignmentManager.this,
> regionState.getRegion()));
> break;
> {code}
> We queue a new close (Should we?).
> We time out a few more times (9 times) and each time we queue a new close.
> Eventually the close succeeds, the region gets assigned a new location.
> Then the next close pops off the eventhandler queue.
> Here is the telltale signature of stuff gone amiss:
> {code}
> 2011-01-05 00:52:19,379 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. state=OPEN, ts=1294188709030
> {code}
> Notice how state is OPEN when we are forcing offline (It was actually just successfully opened). We end up assigning same server because plan was still around:
> {code}
> 2011-01-05 00:52:20,705 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Attempted open of TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. but already online on this server
> {code}
> But later when plan is cleared, we assign new server and we have dbl-assignment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3420) Handling a big rebalance, we can
queue multiple instances of a Close event; messes up state
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977897#action_12977897 ]
stack commented on HBASE-3420:
------------------------------
Looking more, the CLOSED event had been queued over on the master but tens of seconds elapsed before it had a chance to run (This was a rebalance of thousands of regions on constrained server). Meantime, we were requeuing CloseRegionHandlers every ten seconds as the CLOSED timeeout in RIT.
I'm going to post patch that removes the adding new CRH to event queue on timeout of CLOSED. Either the queued original CRH will run or server will crash and region state will be altered appropriately at that time.
> Handling a big rebalance, we can queue multiple instances of a Close event; messes up state
> -------------------------------------------------------------------------------------------
>
> Key: HBASE-3420
> URL: https://issues.apache.org/jira/browse/HBASE-3420
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: stack
> Fix For: 0.90.1
>
>
> This is pretty ugly. In short, on a heavily loaded cluster, we are queuing multiple instances of region close. They all try to run confusing state.
> Long version:
> I have a messy cluster. Its 16k regions on 8 servers. One node has 5k or so regions on it. Heaps are 1G all around. My master had OOME'd. Not sure why but not too worried about it for now. So, new master comes up and is trying to rebalance the cluster:
> {code}
> 2011-01-05 00:48:07,385 INFO org.apache.hadoop.hbase.master.LoadBalancer: Calculated a load balance in 14ms. Moving 3666 regions off of 6 overloaded servers onto 3 less loaded servers
> {code}
> The balancer ends up sending many closes to a single overloaded server are taking so long, the close times out in RIT. We then do this:
> {code}
> case CLOSED:
> LOG.info("Region has been CLOSED for too long, " +
> "retriggering ClosedRegionHandler");
> AssignmentManager.this.executorService.submit(
> new ClosedRegionHandler(master, AssignmentManager.this,
> regionState.getRegion()));
> break;
> {code}
> We queue a new close (Should we?).
> We time out a few more times (9 times) and each time we queue a new close.
> Eventually the close succeeds, the region gets assigned a new location.
> Then the next close pops off the eventhandler queue.
> Here is the telltale signature of stuff gone amiss:
> {code}
> 2011-01-05 00:52:19,379 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. state=OPEN, ts=1294188709030
> {code}
> Notice how state is OPEN when we are forcing offline (It was actually just successfully opened). We end up assigning same server because plan was still around:
> {code}
> 2011-01-05 00:52:20,705 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Attempted open of TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. but already online on this server
> {code}
> But later when plan is cleared, we assign new server and we have dbl-assignment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-3420) Handling a big rebalance, we can
queue multiple instances of a Close event; messes up state
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-3420.
--------------------------
Resolution: Fixed
Fix Version/s: (was: 0.90.1)
0.90.0
Assignee: stack
Hadoop Flags: [Reviewed]
Committed to branch and trunk.
> Handling a big rebalance, we can queue multiple instances of a Close event; messes up state
> -------------------------------------------------------------------------------------------
>
> Key: HBASE-3420
> URL: https://issues.apache.org/jira/browse/HBASE-3420
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: stack
> Assignee: stack
> Fix For: 0.90.0
>
> Attachments: 3420.txt
>
>
> This is pretty ugly. In short, on a heavily loaded cluster, we are queuing multiple instances of region close. They all try to run confusing state.
> Long version:
> I have a messy cluster. Its 16k regions on 8 servers. One node has 5k or so regions on it. Heaps are 1G all around. My master had OOME'd. Not sure why but not too worried about it for now. So, new master comes up and is trying to rebalance the cluster:
> {code}
> 2011-01-05 00:48:07,385 INFO org.apache.hadoop.hbase.master.LoadBalancer: Calculated a load balance in 14ms. Moving 3666 regions off of 6 overloaded servers onto 3 less loaded servers
> {code}
> The balancer ends up sending many closes to a single overloaded server are taking so long, the close times out in RIT. We then do this:
> {code}
> case CLOSED:
> LOG.info("Region has been CLOSED for too long, " +
> "retriggering ClosedRegionHandler");
> AssignmentManager.this.executorService.submit(
> new ClosedRegionHandler(master, AssignmentManager.this,
> regionState.getRegion()));
> break;
> {code}
> We queue a new close (Should we?).
> We time out a few more times (9 times) and each time we queue a new close.
> Eventually the close succeeds, the region gets assigned a new location.
> Then the next close pops off the eventhandler queue.
> Here is the telltale signature of stuff gone amiss:
> {code}
> 2011-01-05 00:52:19,379 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. state=OPEN, ts=1294188709030
> {code}
> Notice how state is OPEN when we are forcing offline (It was actually just successfully opened). We end up assigning same server because plan was still around:
> {code}
> 2011-01-05 00:52:20,705 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Attempted open of TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. but already online on this server
> {code}
> But later when plan is cleared, we assign new server and we have dbl-assignment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3420) Handling a big rebalance, we can
queue multiple instances of a Close event; messes up state
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977930#action_12977930 ]
Jonathan Gray commented on HBASE-3420:
--------------------------------------
It's unrelated to the notion of "checkins" (which is almost completely gone now) so not sure why we would reuse this config param. We could set per-RS limits but that would probably require significantly more hack-up of the balancing algo.
> Handling a big rebalance, we can queue multiple instances of a Close event; messes up state
> -------------------------------------------------------------------------------------------
>
> Key: HBASE-3420
> URL: https://issues.apache.org/jira/browse/HBASE-3420
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: stack
> Fix For: 0.90.1
>
> Attachments: 3420.txt
>
>
> This is pretty ugly. In short, on a heavily loaded cluster, we are queuing multiple instances of region close. They all try to run confusing state.
> Long version:
> I have a messy cluster. Its 16k regions on 8 servers. One node has 5k or so regions on it. Heaps are 1G all around. My master had OOME'd. Not sure why but not too worried about it for now. So, new master comes up and is trying to rebalance the cluster:
> {code}
> 2011-01-05 00:48:07,385 INFO org.apache.hadoop.hbase.master.LoadBalancer: Calculated a load balance in 14ms. Moving 3666 regions off of 6 overloaded servers onto 3 less loaded servers
> {code}
> The balancer ends up sending many closes to a single overloaded server are taking so long, the close times out in RIT. We then do this:
> {code}
> case CLOSED:
> LOG.info("Region has been CLOSED for too long, " +
> "retriggering ClosedRegionHandler");
> AssignmentManager.this.executorService.submit(
> new ClosedRegionHandler(master, AssignmentManager.this,
> regionState.getRegion()));
> break;
> {code}
> We queue a new close (Should we?).
> We time out a few more times (9 times) and each time we queue a new close.
> Eventually the close succeeds, the region gets assigned a new location.
> Then the next close pops off the eventhandler queue.
> Here is the telltale signature of stuff gone amiss:
> {code}
> 2011-01-05 00:52:19,379 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. state=OPEN, ts=1294188709030
> {code}
> Notice how state is OPEN when we are forcing offline (It was actually just successfully opened). We end up assigning same server because plan was still around:
> {code}
> 2011-01-05 00:52:20,705 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Attempted open of TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. but already online on this server
> {code}
> But later when plan is cleared, we assign new server and we have dbl-assignment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3420) Handling a big rebalance, we can
queue multiple instances of a Close event; messes up state
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978156#action_12978156 ]
Hudson commented on HBASE-3420:
-------------------------------
Integrated in HBase-TRUNK #1705 (See [https://hudson.apache.org/hudson/job/HBase-TRUNK/1705/])
> Handling a big rebalance, we can queue multiple instances of a Close event; messes up state
> -------------------------------------------------------------------------------------------
>
> Key: HBASE-3420
> URL: https://issues.apache.org/jira/browse/HBASE-3420
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: stack
> Assignee: stack
> Fix For: 0.90.0
>
> Attachments: 3420.txt
>
>
> This is pretty ugly. In short, on a heavily loaded cluster, we are queuing multiple instances of region close. They all try to run confusing state.
> Long version:
> I have a messy cluster. Its 16k regions on 8 servers. One node has 5k or so regions on it. Heaps are 1G all around. My master had OOME'd. Not sure why but not too worried about it for now. So, new master comes up and is trying to rebalance the cluster:
> {code}
> 2011-01-05 00:48:07,385 INFO org.apache.hadoop.hbase.master.LoadBalancer: Calculated a load balance in 14ms. Moving 3666 regions off of 6 overloaded servers onto 3 less loaded servers
> {code}
> The balancer ends up sending many closes to a single overloaded server are taking so long, the close times out in RIT. We then do this:
> {code}
> case CLOSED:
> LOG.info("Region has been CLOSED for too long, " +
> "retriggering ClosedRegionHandler");
> AssignmentManager.this.executorService.submit(
> new ClosedRegionHandler(master, AssignmentManager.this,
> regionState.getRegion()));
> break;
> {code}
> We queue a new close (Should we?).
> We time out a few more times (9 times) and each time we queue a new close.
> Eventually the close succeeds, the region gets assigned a new location.
> Then the next close pops off the eventhandler queue.
> Here is the telltale signature of stuff gone amiss:
> {code}
> 2011-01-05 00:52:19,379 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. state=OPEN, ts=1294188709030
> {code}
> Notice how state is OPEN when we are forcing offline (It was actually just successfully opened). We end up assigning same server because plan was still around:
> {code}
> 2011-01-05 00:52:20,705 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Attempted open of TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. but already online on this server
> {code}
> But later when plan is cleared, we assign new server and we have dbl-assignment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3420) Handling a big rebalance, we can
queue multiple instances of a Close event; messes up state
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977890#action_12977890 ]
stack commented on HBASE-3420:
------------------------------
Its timeout of a close. Here is sequence:
{code}
2011-01-05 00:49:37,670 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041., src=sv2borg181,60020,1294096110452, dest=sv2borg188,60020,1294187735582
2011-01-05 00:49:37,670 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of region TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. (offlining)
2011-01-05 00:49:37,671 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to serverName=sv2borg181,60020,1294096110452, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) for region TestTable,0487405776,1294125523541. b1fa38bb610943e9eadc604babe4d041.
2011-01-05 00:49:38,310 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x12d3de9e7c60e37 Retrieved 112 byte(s) of data from znode /hbase/unassigned/b1fa38bb610943e9eadc604babe4d041 and set watcher; region=TestTable,0487405776,1294125523541. b1fa38bb610943e9eadc604babe4d041., server=sv2borg181,60020,1294096110452, state=RS_ZK_REGION_CLOSED
2011-01-05 00:49:38,385 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling new unassigned node: /hbase/unassigned/b1fa38bb610943e9eadc604babe4d041 (region=TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041., server=sv2borg181,60020, 1294096110452, state=RS_ZK_REGION_CLOSED)
2011-01-05 00:49:38,385 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_CLOSED, server=sv2borg181,60020,1294096110452, region=b1fa38bb610943e9eadc604babe4d041
2011-01-05 00:50:12,412 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. state=CLOSED, ts=1294188578211
2011-01-05 00:50:12,412 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been CLOSED for too long, retriggering ClosedRegionHandler
{code}
> Handling a big rebalance, we can queue multiple instances of a Close event; messes up state
> -------------------------------------------------------------------------------------------
>
> Key: HBASE-3420
> URL: https://issues.apache.org/jira/browse/HBASE-3420
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: stack
> Fix For: 0.90.1
>
>
> This is pretty ugly. In short, on a heavily loaded cluster, we are queuing multiple instances of region close. They all try to run confusing state.
> Long version:
> I have a messy cluster. Its 16k regions on 8 servers. One node has 5k or so regions on it. Heaps are 1G all around. My master had OOME'd. Not sure why but not too worried about it for now. So, new master comes up and is trying to rebalance the cluster:
> {code}
> 2011-01-05 00:48:07,385 INFO org.apache.hadoop.hbase.master.LoadBalancer: Calculated a load balance in 14ms. Moving 3666 regions off of 6 overloaded servers onto 3 less loaded servers
> {code}
> The balancer ends up sending many closes to a single overloaded server are taking so long, the close times out in RIT. We then do this:
> {code}
> case CLOSED:
> LOG.info("Region has been CLOSED for too long, " +
> "retriggering ClosedRegionHandler");
> AssignmentManager.this.executorService.submit(
> new ClosedRegionHandler(master, AssignmentManager.this,
> regionState.getRegion()));
> break;
> {code}
> We queue a new close (Should we?).
> We time out a few more times (9 times) and each time we queue a new close.
> Eventually the close succeeds, the region gets assigned a new location.
> Then the next close pops off the eventhandler queue.
> Here is the telltale signature of stuff gone amiss:
> {code}
> 2011-01-05 00:52:19,379 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. state=OPEN, ts=1294188709030
> {code}
> Notice how state is OPEN when we are forcing offline (It was actually just successfully opened). We end up assigning same server because plan was still around:
> {code}
> 2011-01-05 00:52:20,705 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Attempted open of TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. but already online on this server
> {code}
> But later when plan is cleared, we assign new server and we have dbl-assignment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3420) Handling a big rebalance, we can
queue multiple instances of a Close event; messes up state
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977906#action_12977906 ]
Jonathan Gray commented on HBASE-3420:
--------------------------------------
So this just updates the timestamp. Seems like it would be equivalent to logging and doing a NO-OP on CLOSED timeout (only point of updating timestamp is to prevent another timeout). I guess this is fine since we will get a log message once per timeout period though.
So once the CRH runs, the RegionState goes to OFFLINE huh? Makes sense then.
+1
and +1 on a maxregionstobalanceatonce or the like
> Handling a big rebalance, we can queue multiple instances of a Close event; messes up state
> -------------------------------------------------------------------------------------------
>
> Key: HBASE-3420
> URL: https://issues.apache.org/jira/browse/HBASE-3420
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: stack
> Fix For: 0.90.1
>
> Attachments: 3420.txt
>
>
> This is pretty ugly. In short, on a heavily loaded cluster, we are queuing multiple instances of region close. They all try to run confusing state.
> Long version:
> I have a messy cluster. Its 16k regions on 8 servers. One node has 5k or so regions on it. Heaps are 1G all around. My master had OOME'd. Not sure why but not too worried about it for now. So, new master comes up and is trying to rebalance the cluster:
> {code}
> 2011-01-05 00:48:07,385 INFO org.apache.hadoop.hbase.master.LoadBalancer: Calculated a load balance in 14ms. Moving 3666 regions off of 6 overloaded servers onto 3 less loaded servers
> {code}
> The balancer ends up sending many closes to a single overloaded server are taking so long, the close times out in RIT. We then do this:
> {code}
> case CLOSED:
> LOG.info("Region has been CLOSED for too long, " +
> "retriggering ClosedRegionHandler");
> AssignmentManager.this.executorService.submit(
> new ClosedRegionHandler(master, AssignmentManager.this,
> regionState.getRegion()));
> break;
> {code}
> We queue a new close (Should we?).
> We time out a few more times (9 times) and each time we queue a new close.
> Eventually the close succeeds, the region gets assigned a new location.
> Then the next close pops off the eventhandler queue.
> Here is the telltale signature of stuff gone amiss:
> {code}
> 2011-01-05 00:52:19,379 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. state=OPEN, ts=1294188709030
> {code}
> Notice how state is OPEN when we are forcing offline (It was actually just successfully opened). We end up assigning same server because plan was still around:
> {code}
> 2011-01-05 00:52:20,705 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Attempted open of TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. but already online on this server
> {code}
> But later when plan is cleared, we assign new server and we have dbl-assignment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3420) Handling a big rebalance, we can
queue multiple instances of a Close event; messes up state
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977929#action_12977929 ]
stack commented on HBASE-3420:
------------------------------
@Ted Balancing works differently in 0.90. Where before, when a RS would heartbeat, in the response we'd give it a set of regions to open/close. The new region assignment goes via zk. The balancer looks at total cluster state and comes up w/ a plan. It then starts the plan rolling which instigates a cascade of closings done via zk.
> Handling a big rebalance, we can queue multiple instances of a Close event; messes up state
> -------------------------------------------------------------------------------------------
>
> Key: HBASE-3420
> URL: https://issues.apache.org/jira/browse/HBASE-3420
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: stack
> Fix For: 0.90.1
>
> Attachments: 3420.txt
>
>
> This is pretty ugly. In short, on a heavily loaded cluster, we are queuing multiple instances of region close. They all try to run confusing state.
> Long version:
> I have a messy cluster. Its 16k regions on 8 servers. One node has 5k or so regions on it. Heaps are 1G all around. My master had OOME'd. Not sure why but not too worried about it for now. So, new master comes up and is trying to rebalance the cluster:
> {code}
> 2011-01-05 00:48:07,385 INFO org.apache.hadoop.hbase.master.LoadBalancer: Calculated a load balance in 14ms. Moving 3666 regions off of 6 overloaded servers onto 3 less loaded servers
> {code}
> The balancer ends up sending many closes to a single overloaded server are taking so long, the close times out in RIT. We then do this:
> {code}
> case CLOSED:
> LOG.info("Region has been CLOSED for too long, " +
> "retriggering ClosedRegionHandler");
> AssignmentManager.this.executorService.submit(
> new ClosedRegionHandler(master, AssignmentManager.this,
> regionState.getRegion()));
> break;
> {code}
> We queue a new close (Should we?).
> We time out a few more times (9 times) and each time we queue a new close.
> Eventually the close succeeds, the region gets assigned a new location.
> Then the next close pops off the eventhandler queue.
> Here is the telltale signature of stuff gone amiss:
> {code}
> 2011-01-05 00:52:19,379 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. state=OPEN, ts=1294188709030
> {code}
> Notice how state is OPEN when we are forcing offline (It was actually just successfully opened). We end up assigning same server because plan was still around:
> {code}
> 2011-01-05 00:52:20,705 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Attempted open of TestTable,0487405776,1294125523541.b1fa38bb610943e9eadc604babe4d041. but already online on this server
> {code}
> But later when plan is cleared, we assign new server and we have dbl-assignment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.