Bare minimum requirements to run a Storm Worker

[Gayashan Amarasinghe <ga...@gmail.com>]

Hi devs,

What are the bare minimum requirements to run a Storm worker on a low
resource environment such as a raspberry pi node where bandwidth
consumption and power consumption could be critical?

Thank you.

Best regards,
/Gayashan

Re: Bare minimum requirements to run a Storm Worker

[Bobby Evans <ev...@oath.com.INVALID>]

Sadly storm is a very talky so reducing the amount of network traffic is
going to be difficult.  I don't think anyone has really tried this before
so you are going to be in new territory.

There are a few things you might want to look at.
1) all of the different daemons in storm tend to poll periodically for
updates.  Almost all of the polling intervals are configurable so you would
want to drop them down.  They probably are not making that much network
traffic, but it could be an easy win.
2) acking does use up a lot of messages.  They are not large, but it is
still a lot of messages.  If you can live with acking off it should reduce
your network load a lot.
3) Try to use the localOrShuffle grouping where ever possible, or even
better just combine your bolts together into a bigger bolts and avoid the
transfer all together.  LocalOrShuffle is a kind a brittle feature when
scheduling does not go perfectly your way and we are working to make it
much better through  https://issues.apache.org/jira/browse/STORM-2684 but
that code is not done yet.

Beyond those I'm not sure what else you would want to do.  You would need
to do some profiling to see what is taking up the memory, network, and CPU.

- Bobby


On Wed, Sep 13, 2017 at 10:36 PM Gayashan Amarasinghe <
gayashan.amarasinghe@gmail.com> wrote:

> Hi Bobby,
>
> Thank you very much for the details. I'm not fixed on a specific version
> yet, would prefer the latest. Is it possible to run with minimum network
> usage, apart from receiving and delivering tuples, such as shutting down
> ackers, running with no supervision, pros and cons of that, etc. Do you
> have any study or details on that or can you point me to where I should
> look at to get some idea?
>
> Thank you.
>
> Best regards,
> /Gayashan
>
> On Tue, Sep 12, 2017 at 12:26 AM, Bobby Evans <ev...@oath.com.invalid>
> wrote:
>
> > A lot of that depends on the version of storm you have, and it is not a
> use
> > case that we have played around with too much.
> >
> > Just looking at a few of our production clusters the CPU usage is very
> > small (but trying to determine what the CPU usage on a raspberry pi arm
> > chip when all you have is the usage on a server CPU is hard to do).  The
> > memory usage, however can be somewhat high, especially for a raspberry
> pi.
> > We are setting the heap size for the supervisor on our clusters to 256MB,
> > but the heap size on the logviewer is 768MB.  But we have not tuned these
> > at all, so it could easily be a lot smaller and still work.
> >
> > - Bobby
> >
> >
> >
> > On Sun, Sep 10, 2017 at 4:36 AM Gayashan Amarasinghe <
> > gayashan.amarasinghe@gmail.com> wrote:
> >
> > > Hi devs,
> > >
> > > What are the bare minimum requirements to run a Storm worker on a low
> > > resource environment such as a raspberry pi node where bandwidth
> > > consumption and power consumption could be critical?
> > >
> > > Thank you.
> > >
> > > Best regards,
> > > /Gayashan
> > >
> >
>

Re: Bare minimum requirements to run a Storm Worker

[Gayashan Amarasinghe <ga...@gmail.com>]

Hi Bobby,

Thank you very much for the details. I'm not fixed on a specific version
yet, would prefer the latest. Is it possible to run with minimum network
usage, apart from receiving and delivering tuples, such as shutting down
ackers, running with no supervision, pros and cons of that, etc. Do you
have any study or details on that or can you point me to where I should
look at to get some idea?

Thank you.

Best regards,
/Gayashan

On Tue, Sep 12, 2017 at 12:26 AM, Bobby Evans <ev...@oath.com.invalid>
wrote:

> A lot of that depends on the version of storm you have, and it is not a use
> case that we have played around with too much.
>
> Just looking at a few of our production clusters the CPU usage is very
> small (but trying to determine what the CPU usage on a raspberry pi arm
> chip when all you have is the usage on a server CPU is hard to do).  The
> memory usage, however can be somewhat high, especially for a raspberry pi.
> We are setting the heap size for the supervisor on our clusters to 256MB,
> but the heap size on the logviewer is 768MB.  But we have not tuned these
> at all, so it could easily be a lot smaller and still work.
>
> - Bobby
>
>
>
> On Sun, Sep 10, 2017 at 4:36 AM Gayashan Amarasinghe <
> gayashan.amarasinghe@gmail.com> wrote:
>
> > Hi devs,
> >
> > What are the bare minimum requirements to run a Storm worker on a low
> > resource environment such as a raspberry pi node where bandwidth
> > consumption and power consumption could be critical?
> >
> > Thank you.
> >
> > Best regards,
> > /Gayashan
> >
>

Re: Bare minimum requirements to run a Storm Worker

[Bobby Evans <ev...@oath.com.INVALID>]

A lot of that depends on the version of storm you have, and it is not a use
case that we have played around with too much.

Just looking at a few of our production clusters the CPU usage is very
small (but trying to determine what the CPU usage on a raspberry pi arm
chip when all you have is the usage on a server CPU is hard to do).  The
memory usage, however can be somewhat high, especially for a raspberry pi.
We are setting the heap size for the supervisor on our clusters to 256MB,
but the heap size on the logviewer is 768MB.  But we have not tuned these
at all, so it could easily be a lot smaller and still work.

- Bobby



On Sun, Sep 10, 2017 at 4:36 AM Gayashan Amarasinghe <
gayashan.amarasinghe@gmail.com> wrote:

> Hi devs,
>
> What are the bare minimum requirements to run a Storm worker on a low
> resource environment such as a raspberry pi node where bandwidth
> consumption and power consumption could be critical?
>
> Thank you.
>
> Best regards,
> /Gayashan
>