auto-added RegionServers

[Norbert Burger <no...@gmail.com>]

We had a situation earlier today in our PROD cluster where a test machine
was accidentally configured with our PROD cluster config.  On startup, the
HMaster promptly accepted the RS into the fold, and started re-assigning
regions to it.  The mass migration caused write latencies to increase, and
it eventually took an HMaster restart to bring things back to normal.

DFS has the dfs.hosts conf setting, which dictates which datanodes are
allowed to join.  In our setup, we're managing dfs.hosts via configuration
management.  From what I can tell looking through
hbase/master/ServerManager.java, there is no equivalent setting on the
HBase side.  Do folks already rely on this auto-add feature, or would it be
helpful if there was a similar stop-gap config param for regionservers?

Norbert

Re: auto-added RegionServers

[Bryan Beaudreault <bb...@hubspot.com>]

We use puppet and ldap to assign cluster names to servers as we spin them in ec2. Our configs are pushed by fabric, which is organized by these cluster names. This makes it harder to assign a server to the wrong hmaster because it will automatically get the configs for the cluster it is in. 

Additionally, we use ec2 security groups to ensure different clusters are sandboxed from each other. You could get this same benefit using VPN or firewall .



Sent from iPhone.

On May 23, 2012, at 11:30 PM, Norbert Burger <no...@gmail.com> wrote:

> We had a situation earlier today in our PROD cluster where a test machine
> was accidentally configured with our PROD cluster config.  On startup, the
> HMaster promptly accepted the RS into the fold, and started re-assigning
> regions to it.  The mass migration caused write latencies to increase, and
> it eventually took an HMaster restart to bring things back to normal.
> 
> DFS has the dfs.hosts conf setting, which dictates which datanodes are
> allowed to join.  In our setup, we're managing dfs.hosts via configuration
> management.  From what I can tell looking through
> hbase/master/ServerManager.java, there is no equivalent setting on the
> HBase side.  Do folks already rely on this auto-add feature, or would it be
> helpful if there was a similar stop-gap config param for regionservers?
> 
> Norbert

Re: auto-added RegionServers

[Michael Drzal <md...@gmail.com>]

I think a similar concept would be a great idea.  It would definitely
prevent the type of issue that you mentioned.  I think that if it was done
in a similar way to how it is handled for hadoop, where you can specify a
list, but if you don't, you get autoadd, should keep everyone happy.

  Mike

On Wed, May 23, 2012 at 8:30 PM, Norbert Burger <no...@gmail.com>wrote:

> We had a situation earlier today in our PROD cluster where a test machine
> was accidentally configured with our PROD cluster config.  On startup, the
> HMaster promptly accepted the RS into the fold, and started re-assigning
> regions to it.  The mass migration caused write latencies to increase, and
> it eventually took an HMaster restart to bring things back to normal.
>
> DFS has the dfs.hosts conf setting, which dictates which datanodes are
> allowed to join.  In our setup, we're managing dfs.hosts via configuration
> management.  From what I can tell looking through
> hbase/master/ServerManager.java, there is no equivalent setting on the
> HBase side.  Do folks already rely on this auto-add feature, or would it be
> helpful if there was a similar stop-gap config param for regionservers?
>
> Norbert
>