Implications of no ackers -- and queue explosion

[Philip O'Toole <ph...@loggly.com>]

This is a question about Storm 0.82.

What are the implications of running without any acking? I realise that
reliability is then not guaranteed, but without ackers I understand that
the "topology.max.spout.pending" doesn't apply. But the Storm docs state
that it is highly-desirable to set this value to prevent "queue explosion".

https://github.com/nathanmarz/storm/wiki/Running-topologies-on-a-production-cluster

Yet the docs also say that by disabling acking, one can improve performance.

https://github.com/nathanmarz/storm/wiki/Guaranteeing-message-processing

So, is running without ackers something to be considered for production
systems? And if so, what throttles the Storm system from flooding the
topology with too many tuples (which I presumes eats up memory, heap,
whatever)?

Thanks,

Philip

Re: Implications of no ackers -- and queue explosion

[Philip O'Toole <ph...@loggly.com>]

On Sun, Nov 10, 2013 at 11:37 AM, Michael Rose <mi...@fullcontact.com>wrote:

> As far as we can tell, if you run without ackers, the Linux OOM killer is
> what eventually prevents Storm from flooding the topology anymore.
>

That'll do it, that's for sure. :-)

Thanks for the feedback.

Philip

Re: Implications of no ackers -- and queue explosion

[Michael Rose <mi...@fullcontact.com>]

As far as we can tell, if you run without ackers, the Linux OOM killer is
what eventually prevents Storm from flooding the topology anymore. ZMQ will
happily accept tuples from upstream and buffer them off-heap until the Java
component can drain the tuples (which may never happen). Same applies for
any kind of topology WITH ackers but with too much geometric tuple growth.

It's true that disabling ackers will improve performance, but at that point
it's kind of just up to you ensuring that downstream components are fast
enough to keep up with upstream traffic. We aren't trying to send 300k
tuple/s/node, so we find the overhead of ackers to be a more than
acceptable cost.

Michael

Michael Rose (@Xorlev <https://twitter.com/xorlev>)
Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
michael@fullcontact.com


On Sun, Nov 10, 2013 at 12:15 PM, Philip O'Toole <ph...@loggly.com> wrote:

> This is a question about Storm 0.82.
>
> What are the implications of running without any acking? I realise that
> reliability is then not guaranteed, but without ackers I understand that
> the "topology.max.spout.pending" doesn't apply. But the Storm docs state
> that it is highly-desirable to set this value to prevent "queue explosion".
>
>
> https://github.com/nathanmarz/storm/wiki/Running-topologies-on-a-production-cluster
>
> Yet the docs also say that by disabling acking, one can improve
> performance.
>
> https://github.com/nathanmarz/storm/wiki/Guaranteeing-message-processing
>
> So, is running without ackers something to be considered for production
> systems? And if so, what throttles the Storm system from flooding the
> topology with too many tuples (which I presumes eats up memory, heap,
> whatever)?
>
> Thanks,
>
> Philip
>