[jira] Commented: (HBASE-2409) Daughters deployed to parents regionserver node, always

["Jonathan Gray (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853571#action_12853571 ] 

Jonathan Gray commented on HBASE-2409:
--------------------------------------

This kind of happens.

It's always been the case that daughters of a split region get assigned back to the original regionserver.  In small clusters without lots of existing regions, this can cause the load balancer to kick in and close the just opened daughters.

The story is kind of worse than that on a small cluster without lots of existing regions.

If you are writing fast enough, it's quite easy to get regions that are 4X or more the split size.  This is because of stuff we're starting to deal with in other issues (HBASE-2375, double flush optimization, etc).  By the time compaction finishes on a loaded cluster, you've managed to snapshot/flush a bunch of additional StoreFiles for that same region.  So compaction finishes, decides to split, we close the region, flush the memstore, split, regions get assigned back, we open them, we compact them (we always compact opened regions when they contain references because we do not support splitting splits).  When the daughters get compacted, we decide to split again.  We close them, flush them, and again they get reassigned back.

We had cases on 5 node clusters where you would see one server hosting root, one server hosting meta, two emtpy servers, and one server with 8 regions.

If you keep the regions in this loop where they decide to split post initial compaction, that regionserver continues to get swamped.

Anyways, we clearly need to have one of the daughters go to a different rs :)  Would also be good if it was not done in a fixed way that could cause the second daughter to always go to the same (different) server.

> Daughters deployed to parents regionserver node, always
> -------------------------------------------------------
>
>                 Key: HBASE-2409
>                 URL: https://issues.apache.org/jira/browse/HBASE-2409
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.4, 0.21.0
>
>
> Ryan apparently said this 6 months ago, that on split, the regions are assigned back to the parent hosting regionserver almost always.  Then, to keep up some kinda balance, the load balancer kicks and closes the just opened daughters -- which have just been opened a moment ago -- to deploy them elsewhere in the name of keeping good balance.
> This issue is to confirm the above behavior indeed happens and then to take action to make it so at least one of the daughters is held up so it doesn't go back to the current heartbeating host, the parent hosting server.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.