launching multiple cluster using whirr

[Samarth Gupta <sa...@gmail.com>]

Hi,

I was planning to use aws to test a haoop application and evaluating whirr
for setting up the cluster. I have tried some whirr recipes, and seen that
each script is starting single cluster. I wanted to know is it possible to
have a recipe launch multiple clusters ?

Below is the kind of set up i would like to have once launch-cluster is
over :

*Cluster A:*

with roles : namenode, datanode, tasktracker, jobtracker, oozie,
MyApp-master

*Cluster B :*

with roles : namenode, datanode, tasktacker, jobtracker, oozie, MyApp-slave


I will be writing whirr services for MyApp-master , MyApp-slave. These
services will require HDFS end points of the cluster A and B. Once of the
uses of application is to replicate data from cluster A to B and vices
versa, using "distcp"  so those details will be needed while MyApp service
is being deployed.

is it possible to have above kind to AWS setup using whirr ?

Thanks in Advance :
Samarth

Re: launching multiple cluster using whirr

[Andrei Savu <sa...@gmail.com>]

Whirr has no support for multiple clusters but you can make this happen by
writing your own thin layer on top. Whirr was designed to be used both as a
standalone tool but also as a library.

Regards,

-- Andrei Savu / axemblr.com


On Wed, Aug 28, 2013 at 6:54 PM, Samarth Gupta <sa...@gmail.com>wrote:

> Hi,
>
> I was planning to use aws to test a haoop application and evaluating whirr
> for setting up the cluster. I have tried some whirr recipes, and seen that
> each script is starting single cluster. I wanted to know is it possible to
> have a recipe launch multiple clusters ?
>
> Below is the kind of set up i would like to have once launch-cluster is
> over :
>
> *Cluster A:*
>
> with roles : namenode, datanode, tasktracker, jobtracker, oozie,
> MyApp-master
>
> *Cluster B :*
>
> with roles : namenode, datanode, tasktacker, jobtracker, oozie, MyApp-slave
>
>
> I will be writing whirr services for MyApp-master , MyApp-slave. These
> services will require HDFS end points of the cluster A and B. Once of the
> uses of application is to replicate data from cluster A to B and vices
> versa, using "distcp"  so those details will be needed while MyApp service
> is being deployed.
>
> is it possible to have above kind to AWS setup using whirr ?
>
> Thanks in Advance :
> Samarth
>