You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Johannes Hugo Kitschke <jo...@rwth-aachen.de> on 2015/04/03 13:39:19 UTC
Storm Cluster + Cluster Queue
Hi,
till now I used the local mode to test my topology. Now I want to use a
'real' storm cluster, but I am a bit overwhelmed concerning the number
of things I have to setup. Before I dig into all of this I'd like to
hear your opinion if my plan is in principle possible: I have access to
the cluster of my university which provides a queue system for larger
jobs (I am aware that this is not the typical use case for Storm, but
atm this is a academic / proof of concept project). So my first idea to
use this:
Run all processes on a single node. Setup and start all necessary
services with a single script. Is this is in principle possible? Maybe
some of you even did this and could give me an example script.
What are your thoughts?
Have a nice day,
Johannes
Re: Storm Cluster + Cluster Queue
Posted by Andrew Xor <an...@gmail.com>.
Well in that case, we must give a shout-out to wirbelstrum from Michael here
https://github.com/miguno/wirbelsturm
Hope this helps.
On Thu, Apr 9, 2015 at 7:24 PM, Tyson Norris <tn...@adobe.com> wrote:
> docker+fig work great
>
> try this: https://github.com/wurstmeister/storm-docker
>
> On Apr 9, 2015, at 6:32 AM, Johannes Hugo Kitschke <
> johannes.kitschke@rwth-aachen.de> wrote:
>
> Thanks for your answer. Do you have, by chance, a startup script at
> hand? Until now I was only using the pyleus local mode, so a start up
> script as reference could reduce my effort.
>
> In may case this may be a bit more difficult since I have no sudo rights.
> To get Storm to run was already a bit complicated. But the info that I only
> need to install zookeeper is a great hint. I'll give it a try.
>
> On 04/03/2015 02:17 PM, Nathan Leung wrote:
>
> I've run all required services on a vm on my desktop. It's fine as long as
> you're not doing anything intensive. You really just need the storm
> processes and zoo keeper so it's not so bad.
> On Apr 3, 2015 7:39 AM, "Johannes Hugo Kitschke" <
> johannes.kitschke@rwth-aachen.de> wrote:
>
>> Hi,
>>
>> till now I used the local mode to test my topology. Now I want to use a
>> 'real' storm cluster, but I am a bit overwhelmed concerning the number of
>> things I have to setup. Before I dig into all of this I'd like to hear your
>> opinion if my plan is in principle possible: I have access to the cluster
>> of my university which provides a queue system for larger jobs (I am aware
>> that this is not the typical use case for Storm, but atm this is a academic
>> / proof of concept project). So my first idea to use this:
>> Run all processes on a single node. Setup and start all necessary
>> services with a single script. Is this is in principle possible? Maybe some
>> of you even did this and could give me an example script.
>>
>> What are your thoughts?
>>
>> Have a nice day,
>> Johannes
>>
>
>
> --
> Johannes Hugo Kitschke
> Am Bollet 6
> D-52078 Aachen
> Mobil: 0177 / 3233 941
>
>
>
Re: Storm Cluster + Cluster Queue
Posted by Tyson Norris <tn...@adobe.com>.
docker+fig work great
try this: https://github.com/wurstmeister/storm-docker
On Apr 9, 2015, at 6:32 AM, Johannes Hugo Kitschke <jo...@rwth-aachen.de>> wrote:
Thanks for your answer. Do you have, by chance, a startup script at hand? Until now I was only using the pyleus local mode, so a start up script as reference could reduce my effort.
In may case this may be a bit more difficult since I have no sudo rights. To get Storm to run was already a bit complicated. But the info that I only need to install zookeeper is a great hint. I'll give it a try.
On 04/03/2015 02:17 PM, Nathan Leung wrote:
I've run all required services on a vm on my desktop. It's fine as long as you're not doing anything intensive. You really just need the storm processes and zoo keeper so it's not so bad.
On Apr 3, 2015 7:39 AM, "Johannes Hugo Kitschke" <jo...@rwth-aachen.de>> wrote:
Hi,
till now I used the local mode to test my topology. Now I want to use a 'real' storm cluster, but I am a bit overwhelmed concerning the number of things I have to setup. Before I dig into all of this I'd like to hear your opinion if my plan is in principle possible: I have access to the cluster of my university which provides a queue system for larger jobs (I am aware that this is not the typical use case for Storm, but atm this is a academic / proof of concept project). So my first idea to use this:
Run all processes on a single node. Setup and start all necessary services with a single script. Is this is in principle possible? Maybe some of you even did this and could give me an example script.
What are your thoughts?
Have a nice day,
Johannes
--
Johannes Hugo Kitschke
Am Bollet 6
D-52078 Aachen
Mobil: 0177 / 3233 941
Re: Storm Cluster + Cluster Queue
Posted by Nathan Leung <nc...@gmail.com>.
I don't recall off the top of my head exactly but it's something like
cd <storm install>
./bin/storm nimbus
./bin/storm ui
./bin/storm supervisor
cd <zookeeper install>
./bin/zkServer.sh start
At least with older versions (I haven't tried with newer) you need to be in
the cwd you want storm to use when you start the service, especially
supervisor.
On Apr 9, 2015 9:33 AM, "Johannes Hugo Kitschke" <
johannes.kitschke@rwth-aachen.de> wrote:
> Thanks for your answer. Do you have, by chance, a startup script at
> hand? Until now I was only using the pyleus local mode, so a start up
> script as reference could reduce my effort.
>
> In may case this may be a bit more difficult since I have no sudo rights.
> To get Storm to run was already a bit complicated. But the info that I only
> need to install zookeeper is a great hint. I'll give it a try.
>
> On 04/03/2015 02:17 PM, Nathan Leung wrote:
>
> I've run all required services on a vm on my desktop. It's fine as long as
> you're not doing anything intensive. You really just need the storm
> processes and zoo keeper so it's not so bad.
> On Apr 3, 2015 7:39 AM, "Johannes Hugo Kitschke" <
> johannes.kitschke@rwth-aachen.de> wrote:
>
>> Hi,
>>
>> till now I used the local mode to test my topology. Now I want to use a
>> 'real' storm cluster, but I am a bit overwhelmed concerning the number of
>> things I have to setup. Before I dig into all of this I'd like to hear your
>> opinion if my plan is in principle possible: I have access to the cluster
>> of my university which provides a queue system for larger jobs (I am aware
>> that this is not the typical use case for Storm, but atm this is a academic
>> / proof of concept project). So my first idea to use this:
>> Run all processes on a single node. Setup and start all necessary
>> services with a single script. Is this is in principle possible? Maybe some
>> of you even did this and could give me an example script.
>>
>> What are your thoughts?
>>
>> Have a nice day,
>> Johannes
>>
>
>
> --
> Johannes Hugo Kitschke
> Am Bollet 6
> D-52078 Aachen
> Mobil: 0177 / 3233 941
>
>
Re: Storm Cluster + Cluster Queue
Posted by Johannes Hugo Kitschke <jo...@rwth-aachen.de>.
Thanks for your answer. Do you have, by chance, a startup script at
hand? Until now I was only using the pyleus local mode, so a start up
script as reference could reduce my effort.
In may case this may be a bit more difficult since I have no sudo
rights. To get Storm to run was already a bit complicated. But the info
that I only need to install zookeeper is a great hint. I'll give it a try.
On 04/03/2015 02:17 PM, Nathan Leung wrote:
>
> I've run all required services on a vm on my desktop. It's fine as
> long as you're not doing anything intensive. You really just need the
> storm processes and zoo keeper so it's not so bad.
>
> On Apr 3, 2015 7:39 AM, "Johannes Hugo Kitschke"
> <johannes.kitschke@rwth-aachen.de
> <ma...@rwth-aachen.de>> wrote:
>
> Hi,
>
> till now I used the local mode to test my topology. Now I want to
> use a 'real' storm cluster, but I am a bit overwhelmed concerning
> the number of things I have to setup. Before I dig into all of
> this I'd like to hear your opinion if my plan is in principle
> possible: I have access to the cluster of my university which
> provides a queue system for larger jobs (I am aware that this is
> not the typical use case for Storm, but atm this is a academic /
> proof of concept project). So my first idea to use this:
> Run all processes on a single node. Setup and start all necessary
> services with a single script. Is this is in principle possible?
> Maybe some of you even did this and could give me an example script.
>
> What are your thoughts?
>
> Have a nice day,
> Johannes
>
--
Johannes Hugo Kitschke
Am Bollet 6
D-52078 Aachen
Mobil: 0177 / 3233 941
Re: Storm Cluster + Cluster Queue
Posted by Nathan Leung <nc...@gmail.com>.
I've run all required services on a vm on my desktop. It's fine as long as
you're not doing anything intensive. You really just need the storm
processes and zoo keeper so it's not so bad.
On Apr 3, 2015 7:39 AM, "Johannes Hugo Kitschke" <
johannes.kitschke@rwth-aachen.de> wrote:
> Hi,
>
> till now I used the local mode to test my topology. Now I want to use a
> 'real' storm cluster, but I am a bit overwhelmed concerning the number of
> things I have to setup. Before I dig into all of this I'd like to hear your
> opinion if my plan is in principle possible: I have access to the cluster
> of my university which provides a queue system for larger jobs (I am aware
> that this is not the typical use case for Storm, but atm this is a academic
> / proof of concept project). So my first idea to use this:
> Run all processes on a single node. Setup and start all necessary services
> with a single script. Is this is in principle possible? Maybe some of you
> even did this and could give me an example script.
>
> What are your thoughts?
>
> Have a nice day,
> Johannes
>