You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Al Hennessey <al...@gmail.com> on 2016/11/22 17:08:53 UTC
Containerising Storm
Hi
I am looking to containerise my storm setup so I can run it across services
like kubernetes or mesos. I can't find an official docker release and
wondering what you guys recommend or whether it's easier just to package
one myself.
Thanks for the help
Re: Containerising Storm
Posted by Xin Wang <da...@gmail.com>.
Hi AI,
You can take a look at the `Container, Resource Management System
Integration` part in https://issues.apache.org/jira/browse/STORM-2209
Thanks,
Xin Wang (vesense)
2016-11-23 16:00 GMT+08:00 <Sa...@swisscom.com>:
> Maybe http://highscalability.com/blog/2016/4/25/the-joy-of-
> deploying-apache-storm-on-docker-swarm.html is also worth having a look
> at?
>
> With kind regards
>
> Samuel
>
> -----Original Message-----
> From: Marc Roos [mailto:M.Roos@f1-outsourcing.eu]
> Sent: Dienstag, 22. November 2016 19:44
> To: user <us...@storm.apache.org>
> Subject: RE: Containerising Storm
>
>
> What about having a look at Heron? Afaik it was developed specifically to
> be run with an external scheduling component.
>
>
>
>
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -.
> F1 Outsourcing Development Sp. z o.o.
> Poland
>
> t: +48 (0)124466845
> f: +48 (0)124466843
> e: marc@f1-outsourcing.eu
>
>
> -----Original Message-----
> From: Zachary Smith [mailto:tud20772@temple.edu]
> Sent: dinsdag 22 november 2016 19:28
> To: user@storm.apache.org
> Subject: Re: Containerising Storm
>
> storm-mesos also provides a way to use storm on top of mesos/marathon
>
> https://github.com/mesos/storm
>
>
> On Tue, Nov 22, 2016 at 12:52 PM, Ambud Sharma <as...@gmail.com>
> wrote:
>
>
> https://github.com/Symantec/hendrix/blob/current/install/
> scripts/lo
> cal-build-install-docker.sh
> <https://github.com/Symantec/hendrix/blob/current/install/
> scripts/local-build-install-docker.sh>
>
> Here's a docker compose way of doing it. Zookeeper in this case is
> not scalable.
>
> I had to rebuild some of the images for Storm to fix issues with
> environment variables and DNS lookups. Those images should work with
> kubernetes as well.
>
>
> On Nov 22, 2016 9:48 AM, "Cuneo, Nicholas" <nc...@tycoint.com>
> wrote:
>
>
> We tried doing the same but ran into deployment issues with
> storm workers being deployed on separate docker instances. I wasn't
> directly involved so I can't give much details, I just know that storm
> is no longer containerized on our end.
>
> ---
> Sent from Boxer | http://getboxer.com <http://bxr.io/PBID1
> >
>
>
> On November 22, 2016 at 9:08:53 AM PST, Al Hennessey
> <al...@gmail.com> wrote:
>
>
> Hi
> I am looking to containerise my storm setup so I
> can run
> it across services like kubernetes or mesos. I can't find an official
> docker release and wondering what you guys recommend or whether it's
> easier just to package one myself.
>
> Thanks for the help
>
>
> ________________________________
>
>
> This e-mail contains privileged and confidential
> information
> intended for the use of the addressees named above. If you are not the
> intended recipient of this e-mail, you are hereby notified that you must
> not disseminate, copy or take any action in respect of any information
> contained in it. If you have received this e-mail in error, please
> notify the sender immediately by e-mail and immediately destroy this
> e-mail and its attachments.
>
>
>
>
>
RE: Containerising Storm
Posted by Sa...@swisscom.com.
Maybe http://highscalability.com/blog/2016/4/25/the-joy-of-deploying-apache-storm-on-docker-swarm.html is also worth having a look at?
With kind regards
Samuel
-----Original Message-----
From: Marc Roos [mailto:M.Roos@f1-outsourcing.eu]
Sent: Dienstag, 22. November 2016 19:44
To: user <us...@storm.apache.org>
Subject: RE: Containerising Storm
What about having a look at Heron? Afaik it was developed specifically to be run with an external scheduling component.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -.
F1 Outsourcing Development Sp. z o.o.
Poland
t: +48 (0)124466845
f: +48 (0)124466843
e: marc@f1-outsourcing.eu
-----Original Message-----
From: Zachary Smith [mailto:tud20772@temple.edu]
Sent: dinsdag 22 november 2016 19:28
To: user@storm.apache.org
Subject: Re: Containerising Storm
storm-mesos also provides a way to use storm on top of mesos/marathon
https://github.com/mesos/storm
On Tue, Nov 22, 2016 at 12:52 PM, Ambud Sharma <as...@gmail.com>
wrote:
https://github.com/Symantec/hendrix/blob/current/install/scripts/lo
cal-build-install-docker.sh
<https://github.com/Symantec/hendrix/blob/current/install/scripts/local-build-install-docker.sh>
Here's a docker compose way of doing it. Zookeeper in this case is
not scalable.
I had to rebuild some of the images for Storm to fix issues with
environment variables and DNS lookups. Those images should work with
kubernetes as well.
On Nov 22, 2016 9:48 AM, "Cuneo, Nicholas" <nc...@tycoint.com>
wrote:
We tried doing the same but ran into deployment issues with
storm workers being deployed on separate docker instances. I wasn't
directly involved so I can't give much details, I just know that storm
is no longer containerized on our end.
---
Sent from Boxer | http://getboxer.com <http://bxr.io/PBID1>
On November 22, 2016 at 9:08:53 AM PST, Al Hennessey
<al...@gmail.com> wrote:
Hi
I am looking to containerise my storm setup so I can run
it across services like kubernetes or mesos. I can't find an official
docker release and wondering what you guys recommend or whether it's
easier just to package one myself.
Thanks for the help
________________________________
This e-mail contains privileged and confidential information
intended for the use of the addressees named above. If you are not the
intended recipient of this e-mail, you are hereby notified that you must
not disseminate, copy or take any action in respect of any information
contained in it. If you have received this e-mail in error, please
notify the sender immediately by e-mail and immediately destroy this
e-mail and its attachments.
RE: Containerising Storm
Posted by Marc Roos <M....@f1-outsourcing.eu>.
What about having a look at Heron? Afaik it was developed specifically
to be run with an external scheduling component.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -.
F1 Outsourcing Development Sp. z o.o.
Poland
t: +48 (0)124466845
f: +48 (0)124466843
e: marc@f1-outsourcing.eu
-----Original Message-----
From: Zachary Smith [mailto:tud20772@temple.edu]
Sent: dinsdag 22 november 2016 19:28
To: user@storm.apache.org
Subject: Re: Containerising Storm
storm-mesos also provides a way to use storm on top of mesos/marathon
https://github.com/mesos/storm
On Tue, Nov 22, 2016 at 12:52 PM, Ambud Sharma <as...@gmail.com>
wrote:
https://github.com/Symantec/hendrix/blob/current/install/scripts/lo
cal-build-install-docker.sh
<https://github.com/Symantec/hendrix/blob/current/install/scripts/local-build-install-docker.sh>
Here's a docker compose way of doing it. Zookeeper in this case is
not scalable.
I had to rebuild some of the images for Storm to fix issues with
environment variables and DNS lookups. Those images should work with
kubernetes as well.
On Nov 22, 2016 9:48 AM, "Cuneo, Nicholas" <nc...@tycoint.com>
wrote:
We tried doing the same but ran into deployment issues with
storm workers being deployed on separate docker instances. I wasn't
directly involved so I can't give much details, I just know that storm
is no longer containerized on our end.
---
Sent from Boxer | http://getboxer.com <http://bxr.io/PBID1>
On November 22, 2016 at 9:08:53 AM PST, Al Hennessey
<al...@gmail.com> wrote:
Hi
I am looking to containerise my storm setup so I can run
it across services like kubernetes or mesos. I can't find an official
docker release and wondering what you guys recommend or whether it's
easier just to package one myself.
Thanks for the help
________________________________
This e-mail contains privileged and confidential information
intended for the use of the addressees named above. If you are not the
intended recipient of this e-mail, you are hereby notified that you must
not disseminate, copy or take any action in respect of any information
contained in it. If you have received this e-mail in error, please
notify the sender immediately by e-mail and immediately destroy this
e-mail and its attachments.
Re: Containerising Storm
Posted by Zachary Smith <tu...@temple.edu>.
storm-mesos also provides a way to use storm on top of mesos/marathon
https://github.com/mesos/storm
On Tue, Nov 22, 2016 at 12:52 PM, Ambud Sharma <as...@gmail.com>
wrote:
> https://github.com/Symantec/hendrix/blob/current/install/
> scripts/local-build-install-docker.sh
>
> Here's a docker compose way of doing it. Zookeeper in this case is not
> scalable.
>
> I had to rebuild some of the images for Storm to fix issues with
> environment variables and DNS lookups. Those images should work with
> kubernetes as well.
>
> On Nov 22, 2016 9:48 AM, "Cuneo, Nicholas" <nc...@tycoint.com> wrote:
>
>> We tried doing the same but ran into deployment issues with storm workers
>> being deployed on separate docker instances. I wasn't directly involved so
>> I can't give much details, I just know that storm is no longer
>> containerized on our end.
>>
>> ---
>> Sent from Boxer | http://getboxer.com <http://bxr.io/PBID1>
>>
>> On November 22, 2016 at 9:08:53 AM PST, Al Hennessey <
>> alhennessey92@gmail.com> wrote:
>>
>> Hi
>> I am looking to containerise my storm setup so I can run it across
>> services like kubernetes or mesos. I can't find an official docker release
>> and wondering what you guys recommend or whether it's easier just to
>> package one myself.
>>
>> Thanks for the help
>>
>>
>> ------------------------------
>>
>> This e-mail contains privileged and confidential information intended for
>> the use of the addressees named above. If you are not the intended
>> recipient of this e-mail, you are hereby notified that you must not
>> disseminate, copy or take any action in respect of any information
>> contained in it. If you have received this e-mail in error, please notify
>> the sender immediately by e-mail and immediately destroy this e-mail and
>> its attachments.
>>
>
Re: Containerising Storm
Posted by Ambud Sharma <as...@gmail.com>.
https://github.com/Symantec/hendrix/blob/current/install/scripts/local-build-install-docker.sh
Here's a docker compose way of doing it. Zookeeper in this case is not
scalable.
I had to rebuild some of the images for Storm to fix issues with
environment variables and DNS lookups. Those images should work with
kubernetes as well.
On Nov 22, 2016 9:48 AM, "Cuneo, Nicholas" <nc...@tycoint.com> wrote:
> We tried doing the same but ran into deployment issues with storm workers
> being deployed on separate docker instances. I wasn't directly involved so
> I can't give much details, I just know that storm is no longer
> containerized on our end.
>
> ---
> Sent from Boxer | http://getboxer.com <http://bxr.io/PBID1>
>
> On November 22, 2016 at 9:08:53 AM PST, Al Hennessey <
> alhennessey92@gmail.com> wrote:
>
> Hi
> I am looking to containerise my storm setup so I can run it across
> services like kubernetes or mesos. I can't find an official docker release
> and wondering what you guys recommend or whether it's easier just to
> package one myself.
>
> Thanks for the help
>
>
> ------------------------------
>
> This e-mail contains privileged and confidential information intended for
> the use of the addressees named above. If you are not the intended
> recipient of this e-mail, you are hereby notified that you must not
> disseminate, copy or take any action in respect of any information
> contained in it. If you have received this e-mail in error, please notify
> the sender immediately by e-mail and immediately destroy this e-mail and
> its attachments.
>
Re: Containerising Storm
Posted by "Cuneo, Nicholas" <nc...@tycoint.com>.
We tried doing the same but ran into deployment issues with storm workers being deployed on separate docker instances. I wasn't directly involved so I can't give much details, I just know that storm is no longer containerized on our end.
---
Sent from Boxer | http://getboxer.com<http://bxr.io/PBID1>
On November 22, 2016 at 9:08:53 AM PST, Al Hennessey <al...@gmail.com> wrote:
Hi
I am looking to containerise my storm setup so I can run it across services like kubernetes or mesos. I can't find an official docker release and wondering what you guys recommend or whether it's easier just to package one myself.
Thanks for the help
________________________________
This e-mail contains privileged and confidential information intended for the use of the addressees named above. If you are not the intended recipient of this e-mail, you are hereby notified that you must not disseminate, copy or take any action in respect of any information contained in it. If you have received this e-mail in error, please notify the sender immediately by e-mail and immediately destroy this e-mail and its attachments.
Re: Containerising Storm
Posted by Paritosh Anand <ka...@gmail.com>.
Hi,
There is an official image for Apache Storm available on Docker hub. You
can start by trying this image.
If you may, you can also try this image
https://hub.docker.com/r/kakapari/docker-storm/ this simply installs
1. Zookeeper v3.4.8
2. Storm nimbus v0.10.1
3. Storm supervisor with 2 workers.
It is easier if you go through the Dockerfile for these repo and you might
find easy to write your own as well.
P
On 22 November 2016 at 22:38, Al Hennessey <al...@gmail.com> wrote:
> Hi
> I am looking to containerise my storm setup so I can run it across
> services like kubernetes or mesos. I can't find an official docker release
> and wondering what you guys recommend or whether it's easier just to
> package one myself.
>
> Thanks for the help
>