Adding new service

[Moritz Kaufmann <mo...@googlemail.com>]

Hi,

I currently try to add support for a new service in Whirr. My goal is to
automatically deploy stratosphere ( http://stratosphere.eu ) to AWS. As it
is using HDFS as its data source and has a similar architecural concept like
hadoop Whirr seemed like a good match. My experience with using Whirr is
minimal as I just discovered it today (never actively used a cloud provider
before).

As a starting point I used the HBase / Hadoop service because they both have
the concept of specialiced nodes. I quickly made some progress and got it to
a point where it would create the correct configuration files on the nodes.
But when browsing through JIRA I discovered WHIRR-266 which goal is to
further refine the deployment (which seems like a very good idea to me - the
concept that the install script would also start e.g. hdfs was confusing in
the beginning). So my questions now are:

   1. Will the new concept as already commited for voldemort and cassandra
   substancially change or could I use those services as a reference?
   2. Is there an interest in integrating the new service for stratosphere
   into whirr and if yes what would be the process (e.g. add a jira ticket?)

If you want more information on stratosphere I am happy to answer any
questions.

Also thanks for developing such a usefull project!

--moritz

PS: All my changes will be pushed to my fork of the whirr project in my
public github repository (https://github.com/mkaufmann/whirr) - There is
nothing currently, but I will try to push a initial working state soon.

Re: Adding new service

[Tom White <to...@gmail.com>]

Hi Moritz,

On Thu, May 19, 2011 at 1:07 PM, Moritz Kaufmann
<mo...@googlemail.com> wrote:
> Hi,
>
> I currently try to add support for a new service in Whirr. My goal is to
> automatically deploy stratosphere ( http://stratosphere.eu ) to AWS. As it
> is using HDFS as its data source and has a similar architecural concept like
> hadoop Whirr seemed like a good match. My experience with using Whirr is
> minimal as I just discovered it today (never actively used a cloud provider
> before).
>
> As a starting point I used the HBase / Hadoop service because they both have
> the concept of specialiced nodes. I quickly made some progress and got it to
> a point where it would create the correct configuration files on the nodes.
> But when browsing through JIRA I discovered WHIRR-266 which goal is to
> further refine the deployment (which seems like a very good idea to me - the
> concept that the install script would also start e.g. hdfs was confusing in
> the beginning). So my questions now are:
>
>   1. Will the new concept as already commited for voldemort and cassandra
>   substancially change or could I use those services as a reference?

I don't think they will substantially change, but you can look at the
patch on https://issues.apache.org/jira/browse/WHIRR-294 as it may be
closer to what you are trying to do. It's not committed yet as there
are still a few outstanding problems in testing.

>   2. Is there an interest in integrating the new service for stratosphere
>   into whirr and if yes what would be the process (e.g. add a jira ticket?)

Yes, you would open a JIRA to add it as a service to Whirr.
Alternatively, you can host it with the Stratosphere code and use
Whirr as a library, which should work just as well.

>
> If you want more information on stratosphere I am happy to answer any
> questions.
>
> Also thanks for developing such a usefull project!

Thanks for your interest!

Cheers,
Tom

>
> --moritz
>
> PS: All my changes will be pushed to my fork of the whirr project in my
> public github repository (https://github.com/mkaufmann/whirr) - There is
> nothing currently, but I will try to push a initial working state soon.
>