Whirr callback in between action phases

[Paul Baclace <pa...@gmail.com>]

I have a use case where I want to start up some services in a particular 
order.  I am using Zookeeper to store cluster-wide properties that need 
to be set before dependent services start.  (A shared dbms that needs 
some specific records set before worker nodes start presents a similar 
situation, for example: extending services/hbase to use MySQL for 
metadata.)  What is the right way to do this in Whirr?  My current ideas 
are numbered below.

Looking at ClusterActionHandlerSupport (action callbacks) and:
https://cwiki.apache.org/confluence/display/WHIRR/Implementing+a+New+Service
which states that phases of a Whirr run are essentially cluster-wide 
synchronization barriers (bootstrap, configuration, and I assume also 
start, stop, cleanup, destroy), but there is no way to specify order 
within a phase; I see that:

1.  There is a way to perform an action afterConfiguration() which 
occurs prior to any calls to beforeStart(), as long as a kind of daemon 
only occurs once per cluster (like a master node for HBase) so these 
dependent actions would only happen once.  This works okay for the 
situation where one MySQL could be started up, a new user and db is 
created, and then the beforeStart() for mysql (if mysql was a Whirr 
service) would not addCommand() for start_mysql since it is already 
running. (Alternatively, mysql could be shutdown and then be restarted, 
extending cluster creation time by some seconds.

The downside of this mechanism is that at least one service start occurs 
in an unexpected place, and it does not work when multiple services must 
be started, such as for multiple Zookeepers.  Assuming the unexpected 
place wart is acceptable, I could:

2.  make Zookeeper init work by implementing a barrier of my own inside 
ZooKeeperClusterActionHandler.afterConfiguration() of Zookeeper() that 
uses a countdown latch to load data (properties) into ZK after all ZK 
instances have started, but before any other service node is started.

"Run Whirr twice" approach:
3.  First use Whirr to provision Zookeeper, and then a separate Whirr 
run simply uses the ZK ensemble, loading property data once into ZK 
afterConfiguration().  This approach unfortunately splits the whirr 
state into 2 clusters, complicating network permissions, clusterIDs, and 
shutdown/termination.

Extend Whirr to:
4.  support more than one whirr.instance-templates (using a numeric 
suffix like .0, .1, etc.) in a single run such that each instance 
template is applied in suffix order in 
ClusterController.launchCluster(ClusterSpec); this would mean only one 
Cluster object would be created and there would be no unexpected place 
wart.  The implementation could either consider the templates to be 
overlayed on each other (same number of commas required, noop is default 
service) for the purposes of all phases except the start phase would 
enforce an ordering.

Any thoughts/suggestions?


Paul

Re: Whirr callback in between action phases

[Paul Baclace <pa...@gmail.com>]

On 20121211 1:39 , Andrei Savu wrote:
> Hi Paul -
>
> I would go for the "Run Whirr multiple times" approach. If you adress the
> following issue you should be able to run all the instances in the same
> security group with unrestricted connectivity.
>
> https://issues.apache.org/jira/browse/WHIRR-336
I added a question about the exact requirements.

>
> And to simplify things you can write a customised launcher that will take
> care of allocating cluster names, executing multiple launchCluster commands
> and shutting down everything as expected.
My plan is to add a new service that can be added to the trunk, so it 
needs to work via the command line.  Is there a way to use a customized 
launcher via standard Whirr command line?


Here is a sketch of a variation on the 4th approach ("Whirr Waves"):

Starting with an example of a two "wave" (avoiding "phase" which already 
has a meaning) cluster:

   whirr.instance-templates.0=1 zookeeper, 2 zookeeper, 4 noop

The .0 templates run like normal during "whirr launch-cluster" and then, 
during the same Whirr run, the .1 template is applied as a modification 
of the same cluster so that new host instances are not allocated:

   whirr.instance-templates.1=1 my-master, 2 my-gateway, 4 my-worker

In the second wave, instance provisioning is inhibited, the 
instance-templates must have the same number of commas and same sequence 
of leading numbers, and a role is only allowed to appear in one wave.

This might seem contrived, but here is another example:

   whirr.instance-templates.0=1 mysql-master+zookeeper, 1 
mysql-master+zookeeper,4 noop
   whirr.instance-templates.1=1 hadoop-namenode+hadoop-jobtracker, 1 
hbase-master+hadoop-secondarynn,4 
hadoop-tasktracker+hadoop-datanode+hbase-regionserver

In the first wave, the two mysql-masters form a multi-master ensemble 
which keeps state information about the setup in the Whirr process (much 
like ZooKeeperCluster.getHosts(cluster) informs services/hbase of the 
quorum).

In the second wave, nodes are not allocated 
(BootstrapClusterAction.doAction() is inhibited if instances already 
exist), but all phases for LaunchClusterCommand, including 
beforeBootstrap() and afterBootstrap() callbacks, are executed.  If 
other whirr.cli.command.*Command are run, they would see a combined 
whirr.instance-templates that works like normal.

Obviously, there would be no reason to limit this to 2 waves, but I do 
not expect more than 10 waves to be useful, so the pattern could enable 
a suffix of "\.[0-9]".

If sufficient state is stored and the number of instances of a service 
template were flexible, this is a stepping stone for a way to support a 
Whirr modify-cluster command that can add services and change number of 
instances implementing a service template.


Paul

>
> -- Andrei Savu / axemblr.com
>
>
>
> On Tue, Dec 11, 2012 at 5:25 AM, Paul Baclace <pa...@gmail.com>wrote:
>
>> I have a use case where I want to start up some services in a particular
>> order.  I am using Zookeeper to store cluster-wide properties that need to
>> be set before dependent services start.  (A shared dbms that needs some
>> specific records set before worker nodes start presents a similar
>> situation, for example: extending services/hbase to use MySQL for
>> metadata.)  What is the right way to do this in Whirr?  My current ideas
>> are numbered below.
>>
>> Looking at ClusterActionHandlerSupport (action callbacks) and:
>> https://cwiki.apache.org/**confluence/display/WHIRR/**
>> Implementing+a+New+Service<https://cwiki.apache.org/confluence/display/WHIRR/Implementing+a+New+Service>
>> which states that phases of a Whirr run are essentially cluster-wide
>> synchronization barriers (bootstrap, configuration, and I assume also
>> start, stop, cleanup, destroy), but there is no way to specify order within
>> a phase; I see that:
>>
>> 1.  There is a way to perform an action afterConfiguration() which occurs
>> prior to any calls to beforeStart(), as long as a kind of daemon only
>> occurs once per cluster (like a master node for HBase) so these dependent
>> actions would only happen once.  This works okay for the situation where
>> one MySQL could be started up, a new user and db is created, and then the
>> beforeStart() for mysql (if mysql was a Whirr service) would not
>> addCommand() for start_mysql since it is already running. (Alternatively,
>> mysql could be shutdown and then be restarted, extending cluster creation
>> time by some seconds.
>>
>> The downside of this mechanism is that at least one service start occurs
>> in an unexpected place, and it does not work when multiple services must be
>> started, such as for multiple Zookeepers.  Assuming the unexpected place
>> wart is acceptable, I could:
>>
>> 2.  make Zookeeper init work by implementing a barrier of my own inside
>> ZooKeeperClusterActionHandler.**afterConfiguration() of Zookeeper() that
>> uses a countdown latch to load data (properties) into ZK after all ZK
>> instances have started, but before any other service node is started.
>>
>> "Run Whirr twice" approach:
>> 3.  First use Whirr to provision Zookeeper, and then a separate Whirr run
>> simply uses the ZK ensemble, loading property data once into ZK
>> afterConfiguration().  This approach unfortunately splits the whirr state
>> into 2 clusters, complicating network permissions, clusterIDs, and
>> shutdown/termination.
>>
>> Extend Whirr to:
>> 4.  support more than one whirr.instance-templates (using a numeric suffix
>> like .0, .1, etc.) in a single run such that each instance template is
>> applied in suffix order in ClusterController.**launchCluster(ClusterSpec);
>> this would mean only one Cluster object would be created and there would be
>> no unexpected place wart.  The implementation could either consider the
>> templates to be overlayed on each other (same number of commas required,
>> noop is default service) for the purposes of all phases except the start
>> phase would enforce an ordering.
>>
>> Any thoughts/suggestions?
>>
>>
>> Paul
>>
>>

Re: Whirr callback in between action phases

[Andrei Savu <sa...@gmail.com>]

Hi Paul -

I would go for the "Run Whirr multiple times" approach. If you adress the
following issue you should be able to run all the instances in the same
security group with unrestricted connectivity.

https://issues.apache.org/jira/browse/WHIRR-336

And to simplify things you can write a customised launcher that will take
care of allocating cluster names, executing multiple launchCluster commands
and shutting down everything as expected.

-- Andrei Savu / axemblr.com



On Tue, Dec 11, 2012 at 5:25 AM, Paul Baclace <pa...@gmail.com>wrote:

> I have a use case where I want to start up some services in a particular
> order.  I am using Zookeeper to store cluster-wide properties that need to
> be set before dependent services start.  (A shared dbms that needs some
> specific records set before worker nodes start presents a similar
> situation, for example: extending services/hbase to use MySQL for
> metadata.)  What is the right way to do this in Whirr?  My current ideas
> are numbered below.
>
> Looking at ClusterActionHandlerSupport (action callbacks) and:
> https://cwiki.apache.org/**confluence/display/WHIRR/**
> Implementing+a+New+Service<https://cwiki.apache.org/confluence/display/WHIRR/Implementing+a+New+Service>
> which states that phases of a Whirr run are essentially cluster-wide
> synchronization barriers (bootstrap, configuration, and I assume also
> start, stop, cleanup, destroy), but there is no way to specify order within
> a phase; I see that:
>
> 1.  There is a way to perform an action afterConfiguration() which occurs
> prior to any calls to beforeStart(), as long as a kind of daemon only
> occurs once per cluster (like a master node for HBase) so these dependent
> actions would only happen once.  This works okay for the situation where
> one MySQL could be started up, a new user and db is created, and then the
> beforeStart() for mysql (if mysql was a Whirr service) would not
> addCommand() for start_mysql since it is already running. (Alternatively,
> mysql could be shutdown and then be restarted, extending cluster creation
> time by some seconds.
>
> The downside of this mechanism is that at least one service start occurs
> in an unexpected place, and it does not work when multiple services must be
> started, such as for multiple Zookeepers.  Assuming the unexpected place
> wart is acceptable, I could:
>
> 2.  make Zookeeper init work by implementing a barrier of my own inside
> ZooKeeperClusterActionHandler.**afterConfiguration() of Zookeeper() that
> uses a countdown latch to load data (properties) into ZK after all ZK
> instances have started, but before any other service node is started.
>
> "Run Whirr twice" approach:
> 3.  First use Whirr to provision Zookeeper, and then a separate Whirr run
> simply uses the ZK ensemble, loading property data once into ZK
> afterConfiguration().  This approach unfortunately splits the whirr state
> into 2 clusters, complicating network permissions, clusterIDs, and
> shutdown/termination.
>
> Extend Whirr to:
> 4.  support more than one whirr.instance-templates (using a numeric suffix
> like .0, .1, etc.) in a single run such that each instance template is
> applied in suffix order in ClusterController.**launchCluster(ClusterSpec);
> this would mean only one Cluster object would be created and there would be
> no unexpected place wart.  The implementation could either consider the
> templates to be overlayed on each other (same number of commas required,
> noop is default service) for the purposes of all phases except the start
> phase would enforce an ordering.
>
> Any thoughts/suggestions?
>
>
> Paul
>
>