Running multiple HBase masters / region-servers

[Sean Mackrory <ma...@gmail.com>]

All,

In https://issues.apache.org/jira/browse/BIGTOP-503, several scripts
were removed because (among other things) they circumvented the logic
of the init scripts and thus could cause problems It was recently
requested by some of the HBase folks that local-regionservers.sh and
local-master-backup.sh be re-included. These scripts make it possible
to run multiple region servers and masters on the same system. The
benefits cited include:

* Easy testing of fail-over between nodes on a single system
* Maximizing disk usage with multiple region servers
* Avoiding problems with heaps being too big

The main drawback is the added complexity in the init scripts required
to reliably and clearly deal with PID files, etc... in such a
configuration. I support the rationale behind removing these scripts,
but I also think it'd be nice to eventually support these features in
a safer way through the init scripts added by Bigtop.

I've had a brief look at how a similar thing is done with Apache
HTTPD, but that doesn't seem to apply very well to how these scripts
were working. The design I have in mind is to add a second init script
to the hbase-master and hbase-regionserver packages that is used for
starting and stopping multiple nodes. The status-checks in each script
would then be modified to be aware of whatever the other script may
have done. I envision something like this:

$ service hbase-master # Start 1 master
$ service hbase-multi-master start 0 1 # Attempt to start 2 other masters
Master server already running. Use 'service hbase-master' to administer it.
$ service hbase-multi-regionserver start 0 1 2 3 4 5 6 7 8 9 # Start
10 regionserver
$ service hbase-multi-regionserver stop 0 2 4 6 8 # Stop even region servers
$ service hbase-regionserver status
Multiple region servers running. Use 'service
hbase-multi-regionserver' to administer them.
$ service hbase-multi-regionserver status
Multiple region servers running on offsets: 1, 3, 5, 7, 9

Of course, all existing commands would continue to work the same way,
and perhaps we could even include the "multi" init scripts. in a
separate package to hide the complexity from the people who don't need
to see it.

Just wanted to get some community feedback before embarking on such a
potentially dangerous change... Thoughts?

Re: Running multiple HBase masters / region-servers

[Bruno Mahé <bm...@apache.org>]

On 09/07/2012 02:04 PM, Sean Mackrory wrote:
> All,
>
> In https://issues.apache.org/jira/browse/BIGTOP-503, several scripts
> were removed because (among other things) they circumvented the logic
> of the init scripts and thus could cause problems It was recently
> requested by some of the HBase folks that local-regionservers.sh and
> local-master-backup.sh be re-included. These scripts make it possible
> to run multiple region servers and masters on the same system. The
> benefits cited include:
>
> * Easy testing of fail-over between nodes on a single system
> * Maximizing disk usage with multiple region servers
> * Avoiding problems with heaps being too big
>
> The main drawback is the added complexity in the init scripts required
> to reliably and clearly deal with PID files, etc... in such a
> configuration. I support the rationale behind removing these scripts,
> but I also think it'd be nice to eventually support these features in
> a safer way through the init scripts added by Bigtop.
>
> I've had a brief look at how a similar thing is done with Apache
> HTTPD, but that doesn't seem to apply very well to how these scripts
> were working. The design I have in mind is to add a second init script
> to the hbase-master and hbase-regionserver packages that is used for
> starting and stopping multiple nodes. The status-checks in each script
> would then be modified to be aware of whatever the other script may
> have done. I envision something like this:
>
> $ service hbase-master # Start 1 master
> $ service hbase-multi-master start 0 1 # Attempt to start 2 other masters
> Master server already running. Use 'service hbase-master' to administer it.
> $ service hbase-multi-regionserver start 0 1 2 3 4 5 6 7 8 9 # Start
> 10 regionserver
> $ service hbase-multi-regionserver stop 0 2 4 6 8 # Stop even region servers
> $ service hbase-regionserver status
> Multiple region servers running. Use 'service
> hbase-multi-regionserver' to administer them.
> $ service hbase-multi-regionserver status
> Multiple region servers running on offsets: 1, 3, 5, 7, 9
>
> Of course, all existing commands would continue to work the same way,
> and perhaps we could even include the "multi" init scripts. in a
> separate package to hide the complexity from the people who don't need
> to see it.
>
> Just wanted to get some community feedback before embarking on such a
> potentially dangerous change... Thoughts?
>

This seems reasonable to me. The only issue I have is toward how 
compliant with the LSB this would be (see 
http://refspecs.linuxfoundation.org/LSB_3.1.1/LSB-Core-generic/LSB-Core-generic/iniscrptact.html 
).
So instead of passing the number of instances directly to the script, 
why not reading this from a config file by default? So I could also 
easily restart multiple instances on a reboot of my server.
It does not prevent the service scripts from taking additional 
parameters (ex: I only want to stop the instance 4 and 6), but by 
default, it should just work out of the box.

Also I know that dealing with numbers is easier, but do you see a case 
for being able to have user-defined instance names?

But great idea! I love it.

Thanks,
Bruno