You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Spico Florin <sp...@gmail.com> on 2014/03/26 11:01:25 UTC
Can be storm-deploy script used for deploying storm 0.9.x version on Amazon?
Hello!
I would like to know what changes should be applied to the storm-deploy
script (https://github.com/nathanmarz/storm-deploy) in order to install it
on Amazon?
Thank you in advance.
Regards,
Florin
Re: Can be storm-deploy script used for deploying storm 0.9.x version
on Amazon?
Posted by Yi Wang <e...@yi-wang.me>.
current storm-deply script currently points to
https://github.com/nathanmarz/storm
0.9.0.1 is the latest tag that works
-
Yi
-
Yi
On Wed, Mar 26, 2014 at 6:01 AM, Spico Florin <sp...@gmail.com> wrote:
> Hello!
> I would like to know what changes should be applied to the storm-deploy
> script (https://github.com/nathanmarz/storm-deploy) in order to install
> it on Amazon?
> Thank you in advance.
> Regards,
> Florin
>
Re: Can be storm-deploy script used for deploying storm 0.9.x version
on Amazon?
Posted by "Michael G. Noll" <mi...@michael-noll.com>.
On 03/28/2014 01:46 PM, Spico Florin wrote:
> Hello, all!
> Thank you very much for your helpful responses. Regarding Wirbelsturm,
> even it looks quite appealing at the first glance, in my opinion it is a
> little bit complex
> (https://github.com/miguno/wirbelsturm/blob/master/docs/AWS.md) than the
> initial storm-deploy. It is still in beta version and there are too many
> steps that should be performed for configuring. Also you have to to
> install ec2 cli utilities.
I agree -- the UX to deploy to AWS can still be improved. There are
currently two main reasons why the latest version of Wirbelsturm cannot
yet deploy to AWS in the same simple way as you can for local deployments:
1. A known Vagrant bug that causes problems with stock Amazon Linux AMI.
2. Vagrant's AWS support in general, notably the lack of options to
configure DNS/networking on AWS as easily as you can with local
deployments (for e.g. VirtualBox).
I am looking into ways to address both 1 and 2 above.
Regarding #1, the most recent versions of Vagrant have a new
SSH-related setting that may address the Amazon Linux AMI compatibility
problem. Unfortunately this new feature is poorly documented, and when
I gave it a first try it didn't help to fix bug #1.
Regarding #2, there has been progress on Vagrant plugins such as
vagrant-hostmanager (which I am currently test-driving) that may allow
us to configure DNS related settings on deployed machines without having
to go through Amazon Route 53 as we do currently.
The main drawback of Route 53 is that it requires additional, one-time
configuration steps for Vagrant (and thus Wirbelsturm) because Route 53
e.g. requires delegation of a DNS sub-domain to Route 53 (and for good
reasons such as security you must explicitly enable such delegation for
your DNS domains).
The benefit of Route 53 however is that any additional machines you
run alongside Storm & friends will also learn about the machines you
have deployed via Wirbelsturm/Vagrant. That's nice if you already have
existing infrastructure on AWS. Hence one idea that I have is to make
Route 53 an optional configuration, as long as we can find another way
to configure DNS etc. on AWS instances without Route 53 (such as I hope
we can do with vagrant-hostmanager).
I am just writing this to let you know what the current status of
Wirbelsturm is, and what the plans are moving forward. The AWS user
experience is definitely on the list of things to improve, as is adding
OpenStack support.
> If you have successfully used Wirbelsturm on Windows environment ,
> please let me know how complex was for you.
There are no plans to support Wirbelsturm on Windows -- neither as
host machines nor as guest machines.
Best,
Michael
Re: Can be storm-deploy script used for deploying storm 0.9.x version
on Amazon?
Posted by Spico Florin <sp...@gmail.com>.
Hello, all!
Thank you very much for your helpful responses. Regarding Wirbelsturm,
even it looks quite appealing at the first glance, in my opinion it is a
little bit complex (
https://github.com/miguno/wirbelsturm/blob/master/docs/AWS.md) than the
initial storm-deploy. It is still in beta version and there are too many
steps that should be performed for configuring. Also you have to to install
ec2 cli utilities.
Since I would like something to deploy my application fast, I will go
firstly with storm-deploy but I'll take into the consideration the
Wirbelsturm.
If you have successfully used Wirbelsturm on Windows environment , please
let me know how complex was for you.
Thank you.
Best regards,
Florin
On Fri, Mar 28, 2014 at 12:46 AM, Sasi Panja <sa...@gmail.com> wrote:
> Hi Florin,
> The cluster consisted of 1 master, 1 zookeeper, and 5 workers (test env)
> and it took about 10 minutes for everything to be up and view the UI
> console. The storm-deploy script was run from another ec2 machine running
> in the same zone.
> I was then able to run my topology in less than 2 minutes.
>
> As Marc mentioned, it might be worth to try Michael's Wirbelsturm script,
> which seems quite elegant and powerful from the github/blog documentation.
> -Sasi
>
>
> On Thu, Mar 27, 2014 at 4:15 AM, Marc Vaillant <va...@animetrics.com>wrote:
>
>> Hi Florin,
>>
>> I just wanted to suggest that you also look at Wirbelsturm by Michael
>> Noll as an alternative to storm-deploy
>> https://github.com/miguno/wirbelsturm. I think that you will find it
>> more complete, better documented, and more mainstream because it uses
>> vagrant and puppet instead of pallet and jclouds. I highly recommend
>> reading Michael's blog post about it:
>>
>> http://www.michael-noll.com/blog/2014/03/17/wirbelsturm-one-click-deploy-storm-kafka-clusters-with-vagrant-puppet
>>
>> Best,
>> Marc
>>
>> On Thu, Mar 27, 2014 at 10:19:24AM +0200, Spico Florin wrote:
>> > Hello, Sasi!
>> > Thank your very much for your response.Due to the fact that I would
>> like to
>> > deploy also a Storm application on Amazon, it really helps to spare
>> time . If
>> > it possible, just as an information, do you have some measurement on
>> how much
>> > time did take the cluster to be up and running?
>> >
>> > Best regards,
>> > Florin
>> >
>> >
>> > On Wed, Mar 26, 2014 at 8:36 PM, Sasi Panja <sa...@gmail.com>
>> wrote:
>> >
>> >
>> > I was able to deploy and run 0.9.0-rc2 on EC2 using
>> https://github.com/
>> > nathanmarz/storm-deploy
>> >
>> > After following the instructions from the wiki and updating the
>> > configuration files, the following command worked for me :
>> >
>> > lein deploy-storm --start --name yourclustername --branch master
>> --commit
>> > 0.9.0-rc2
>> >
>> > I am using Leiningen 2.3.4
>> >
>> > All the workers, master and zookkeeper servers were up and running
>> in the
>> > cluster
>> >
>> >
>> ###########################################################################
>> > #####
>> >
>> > # CLUSTERS CONFIG FILE under storm-deploy/conf
>> >
>> >
>> ###########################################################################
>> > #####
>> >
>> > nimbus.image: "us-west-2/ami-ca2ca4fa" #64-bit ubuntu
>> > nimbus.hardware: "m1.xlarge"
>> >
>> > supervisor.count: 5
>> > supervisor.image: "us-west-2/ami-ca2ca4fa" #64-bit ubuntu on
>> > us-west-2
>> > supervisor.hardware: "m1.xlarge"
>> >
>> > zookeeper.count: 1
>> > zookeeper.image: "us-west-2/ami-ca2ca4fa" #64-bit ubuntu
>> > zookeeper.hardware: "m1.large"
>> >
>> >
>> ###########################################################################
>> > #####
>> >
>> > config.clj under ~/.pallet
>> >
>> >
>> ###########################################################################
>> > #####
>> >
>> > (defpallet
>> >
>> > :services
>> >
>> > {
>> >
>> > :default {
>> >
>> > :blobstore-provider "aws-s3"
>> >
>> > :provider "aws-ec2"
>> >
>> > :environment {:user {:username "storm" ; this must be
>> "storm"
>> >
>> > :private-key-path
>> "/home/ubuntu/.ssh/
>> > id_rsa"
>> >
>> > :public-key-path
>> "/home/ubuntu/.ssh/
>> > id_rsa.pub"}
>> >
>> > :aws-user-id "XXXXXXXX}
>> >
>> > :identity "XXXX"
>> >
>> > :credential "XXXXXX"
>> >
>> > :jclouds.regions "us-west-2"
>> >
>> > }
>> >
>> > })
>> >
>> > ----------------------------------------------------------------
>> >
>> >
>> > -Sasi
>> >
>> >
>> > On Wed, Mar 26, 2014 at 3:01 AM, Spico Florin <
>> spicoflorin@gmail.com>
>> > wrote:
>> >
>> > Hello!
>> > I would like to know what changes should be applied to the
>> > storm-deploy script (https://github.com/nathanmarz/storm-deploy)
>> in
>> > order to install it on Amazon?
>> > Thank you in advance.
>> > Regards,
>> > Florin
>> >
>> >
>> >
>> >
>>
>
>
Re: Can be storm-deploy script used for deploying storm 0.9.x version
on Amazon?
Posted by Sasi Panja <sa...@gmail.com>.
Hi Florin,
The cluster consisted of 1 master, 1 zookeeper, and 5 workers (test env)
and it took about 10 minutes for everything to be up and view the UI
console. The storm-deploy script was run from another ec2 machine running
in the same zone.
I was then able to run my topology in less than 2 minutes.
As Marc mentioned, it might be worth to try Michael's Wirbelsturm script,
which seems quite elegant and powerful from the github/blog documentation.
-Sasi
On Thu, Mar 27, 2014 at 4:15 AM, Marc Vaillant <va...@animetrics.com>wrote:
> Hi Florin,
>
> I just wanted to suggest that you also look at Wirbelsturm by Michael
> Noll as an alternative to storm-deploy
> https://github.com/miguno/wirbelsturm. I think that you will find it
> more complete, better documented, and more mainstream because it uses
> vagrant and puppet instead of pallet and jclouds. I highly recommend
> reading Michael's blog post about it:
>
> http://www.michael-noll.com/blog/2014/03/17/wirbelsturm-one-click-deploy-storm-kafka-clusters-with-vagrant-puppet
>
> Best,
> Marc
>
> On Thu, Mar 27, 2014 at 10:19:24AM +0200, Spico Florin wrote:
> > Hello, Sasi!
> > Thank your very much for your response.Due to the fact that I would
> like to
> > deploy also a Storm application on Amazon, it really helps to spare time
> . If
> > it possible, just as an information, do you have some measurement on how
> much
> > time did take the cluster to be up and running?
> >
> > Best regards,
> > Florin
> >
> >
> > On Wed, Mar 26, 2014 at 8:36 PM, Sasi Panja <sa...@gmail.com>
> wrote:
> >
> >
> > I was able to deploy and run 0.9.0-rc2 on EC2 using
> https://github.com/
> > nathanmarz/storm-deploy
> >
> > After following the instructions from the wiki and updating the
> > configuration files, the following command worked for me :
> >
> > lein deploy-storm --start --name yourclustername --branch master
> --commit
> > 0.9.0-rc2
> >
> > I am using Leiningen 2.3.4
> >
> > All the workers, master and zookkeeper servers were up and running
> in the
> > cluster
> >
> >
> ###########################################################################
> > #####
> >
> > # CLUSTERS CONFIG FILE under storm-deploy/conf
> >
> >
> ###########################################################################
> > #####
> >
> > nimbus.image: "us-west-2/ami-ca2ca4fa" #64-bit ubuntu
> > nimbus.hardware: "m1.xlarge"
> >
> > supervisor.count: 5
> > supervisor.image: "us-west-2/ami-ca2ca4fa" #64-bit ubuntu on
> > us-west-2
> > supervisor.hardware: "m1.xlarge"
> >
> > zookeeper.count: 1
> > zookeeper.image: "us-west-2/ami-ca2ca4fa" #64-bit ubuntu
> > zookeeper.hardware: "m1.large"
> >
> >
> ###########################################################################
> > #####
> >
> > config.clj under ~/.pallet
> >
> >
> ###########################################################################
> > #####
> >
> > (defpallet
> >
> > :services
> >
> > {
> >
> > :default {
> >
> > :blobstore-provider "aws-s3"
> >
> > :provider "aws-ec2"
> >
> > :environment {:user {:username "storm" ; this must be
> "storm"
> >
> > :private-key-path
> "/home/ubuntu/.ssh/
> > id_rsa"
> >
> > :public-key-path
> "/home/ubuntu/.ssh/
> > id_rsa.pub"}
> >
> > :aws-user-id "XXXXXXXX}
> >
> > :identity "XXXX"
> >
> > :credential "XXXXXX"
> >
> > :jclouds.regions "us-west-2"
> >
> > }
> >
> > })
> >
> > ----------------------------------------------------------------
> >
> >
> > -Sasi
> >
> >
> > On Wed, Mar 26, 2014 at 3:01 AM, Spico Florin <spicoflorin@gmail.com
> >
> > wrote:
> >
> > Hello!
> > I would like to know what changes should be applied to the
> > storm-deploy script (https://github.com/nathanmarz/storm-deploy)
> in
> > order to install it on Amazon?
> > Thank you in advance.
> > Regards,
> > Florin
> >
> >
> >
> >
>
Re: Can be storm-deploy script used for deploying storm 0.9.x
version on Amazon?
Posted by Marc Vaillant <va...@animetrics.com>.
Hi Florin,
I just wanted to suggest that you also look at Wirbelsturm by Michael
Noll as an alternative to storm-deploy
https://github.com/miguno/wirbelsturm. I think that you will find it
more complete, better documented, and more mainstream because it uses
vagrant and puppet instead of pallet and jclouds. I highly recommend
reading Michael's blog post about it:
http://www.michael-noll.com/blog/2014/03/17/wirbelsturm-one-click-deploy-storm-kafka-clusters-with-vagrant-puppet
Best,
Marc
On Thu, Mar 27, 2014 at 10:19:24AM +0200, Spico Florin wrote:
> Hello, Sasi!
> Thank your very much for your response.Due to the fact that I would like to
> deploy also a Storm application on Amazon, it really helps to spare time . If
> it possible, just as an information, do you have some measurement on how much
> time did take the cluster to be up and running?
>
> Best regards,
> Florin
>
>
> On Wed, Mar 26, 2014 at 8:36 PM, Sasi Panja <sa...@gmail.com> wrote:
>
>
> I was able to deploy and run 0.9.0-rc2 on EC2 using https://github.com/
> nathanmarz/storm-deploy
>
> After following the instructions from the wiki and updating the
> configuration files, the following command worked for me :
>
> lein deploy-storm --start --name yourclustername --branch master --commit
> 0.9.0-rc2
>
> I am using Leiningen 2.3.4
>
> All the workers, master and zookkeeper servers were up and running in the
> cluster
>
> ###########################################################################
> #####
>
> # CLUSTERS CONFIG FILE under storm-deploy/conf
>
> ###########################################################################
> #####
>
> nimbus.image: "us-west-2/ami-ca2ca4fa" #64-bit ubuntu
> nimbus.hardware: "m1.xlarge"
>
> supervisor.count: 5
> supervisor.image: "us-west-2/ami-ca2ca4fa" #64-bit ubuntu on
> us-west-2
> supervisor.hardware: "m1.xlarge"
>
> zookeeper.count: 1
> zookeeper.image: "us-west-2/ami-ca2ca4fa" #64-bit ubuntu
> zookeeper.hardware: "m1.large"
>
> ###########################################################################
> #####
>
> config.clj under ~/.pallet
>
> ###########################################################################
> #####
>
> (defpallet
>
> :services
>
> {
>
> :default {
>
> :blobstore-provider "aws-s3"
>
> :provider "aws-ec2"
>
> :environment {:user {:username "storm" ; this must be "storm"
>
> :private-key-path "/home/ubuntu/.ssh/
> id_rsa"
>
> :public-key-path "/home/ubuntu/.ssh/
> id_rsa.pub"}
>
> :aws-user-id "XXXXXXXX}
>
> :identity "XXXX"
>
> :credential "XXXXXX"
>
> :jclouds.regions "us-west-2"
>
> }
>
> })
>
> ----------------------------------------------------------------
>
>
> -Sasi
>
>
> On Wed, Mar 26, 2014 at 3:01 AM, Spico Florin <sp...@gmail.com>
> wrote:
>
> Hello!
> I would like to know what changes should be applied to the
> storm-deploy script (https://github.com/nathanmarz/storm-deploy) in
> order to install it on Amazon?
> Thank you in advance.
> Regards,
> Florin
>
>
>
>
Re: Can be storm-deploy script used for deploying storm 0.9.x version
on Amazon?
Posted by Spico Florin <sp...@gmail.com>.
Hello, Sasi!
Thank your very much for your response.Due to the fact that I would like
to deploy also a Storm application on Amazon, it really helps to spare time
. If it possible, just as an information, do you have some measurement on
how much time did take the cluster to be up and running?
Best regards,
Florin
On Wed, Mar 26, 2014 at 8:36 PM, Sasi Panja <sa...@gmail.com> wrote:
>
> I was able to deploy and run 0.9.0-rc2 on EC2 using
> https://github.com/nathanmarz/storm-deploy
>
> After following the instructions from the wiki and updating the
> configuration files, the following command worked for me :
>
> *lein deploy-storm --start --name yourclustername --branch master
> --commit 0.9.0-rc2*
>
> I am using Leiningen 2.3.4
>
> All the workers, master and zookkeeper servers were up and running in the
> cluster
>
>
> ################################################################################
>
> # CLUSTERS CONFIG FILE *under storm-deploy/conf*
>
>
> ################################################################################
> nimbus.image: "us-west-2/ami-ca2ca4fa" #64-bit ubuntu
> nimbus.hardware: "m1.xlarge"
>
> supervisor.count: 5
> supervisor.image: "us-west-2/ami-ca2ca4fa" #64-bit ubuntu on
> us-west-2
> supervisor.hardware: "m1.xlarge"
>
> zookeeper.count: 1
> zookeeper.image: "us-west-2/ami-ca2ca4fa" #64-bit ubuntu
> zookeeper.hardware: "m1.large"
>
>
> ################################################################################
> *config.clj under ~/.pallet *
>
>
> ################################################################################
>
> (defpallet
>
> :services
>
> {
>
> :default {
>
> :blobstore-provider "aws-s3"
>
> :provider "aws-ec2"
>
> :environment {:user {:username "storm" ; this must be "storm"
>
> :private-key-path
> "/home/ubuntu/.ssh/id_rsa"
>
> :public-key-path
> "/home/ubuntu/.ssh/id_rsa.pub"}
>
> :aws-user-id "XXXXXXXX}
>
> :identity "XXXX"
>
> :credential "XXXXXX"
>
> :jclouds.regions "us-west-2"
>
> }
>
> })
> ----------------------------------------------------------------
>
>
> -Sasi
>
>
> On Wed, Mar 26, 2014 at 3:01 AM, Spico Florin <sp...@gmail.com>wrote:
>
>> Hello!
>> I would like to know what changes should be applied to the storm-deploy
>> script (https://github.com/nathanmarz/storm-deploy) in order to install
>> it on Amazon?
>> Thank you in advance.
>> Regards,
>> Florin
>>
>
>
Re: Can be storm-deploy script used for deploying storm 0.9.x version
on Amazon?
Posted by Sasi Panja <sa...@gmail.com>.
I was able to deploy and run 0.9.0-rc2 on EC2 using
https://github.com/nathanmarz/storm-deploy
After following the instructions from the wiki and updating the
configuration files, the following command worked for me :
*lein deploy-storm --start --name yourclustername --branch master
--commit 0.9.0-rc2*
I am using Leiningen 2.3.4
All the workers, master and zookkeeper servers were up and running in the
cluster
################################################################################
# CLUSTERS CONFIG FILE *under storm-deploy/conf*
################################################################################
nimbus.image: "us-west-2/ami-ca2ca4fa" #64-bit ubuntu
nimbus.hardware: "m1.xlarge"
supervisor.count: 5
supervisor.image: "us-west-2/ami-ca2ca4fa" #64-bit ubuntu on
us-west-2
supervisor.hardware: "m1.xlarge"
zookeeper.count: 1
zookeeper.image: "us-west-2/ami-ca2ca4fa" #64-bit ubuntu
zookeeper.hardware: "m1.large"
################################################################################
*config.clj under ~/.pallet *
################################################################################
(defpallet
:services
{
:default {
:blobstore-provider "aws-s3"
:provider "aws-ec2"
:environment {:user {:username "storm" ; this must be "storm"
:private-key-path
"/home/ubuntu/.ssh/id_rsa"
:public-key-path
"/home/ubuntu/.ssh/id_rsa.pub"}
:aws-user-id "XXXXXXXX}
:identity "XXXX"
:credential "XXXXXX"
:jclouds.regions "us-west-2"
}
})
----------------------------------------------------------------
-Sasi
On Wed, Mar 26, 2014 at 3:01 AM, Spico Florin <sp...@gmail.com> wrote:
> Hello!
> I would like to know what changes should be applied to the storm-deploy
> script (https://github.com/nathanmarz/storm-deploy) in order to install
> it on Amazon?
> Thank you in advance.
> Regards,
> Florin
>