You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by John Yost <ho...@gmail.com> on 2015/12/23 15:31:50 UTC
topology.ackers set to 0, still acking in downstream bolts?
Hi Everyone,
I have a topology that consists of a KafkaSpout and three downstream bolts.
I turned off acking by setting topology.acker.executors to 0. It's my
understanding that I should only have acking in the KafkaSpout. However, I
see acking in all of my downstream bolts as well. Any ideas as to why?
Thanks
--John
Re: topology.ackers set to 0, still acking in downstream bolts?
Posted by "Matthias J. Sax" <mj...@apache.org>.
Hi John,
fair enough -- I was not aware that you can disable fault-tolerance this
way. Good to know :)
My answer is still the same. Calls to Collector.ack(...) are always
counted and show up in the UI, but *no* ack messages will be sent for
your case.
-Matthias
On 01/04/2016 02:10 AM, John Yost wrote:
> Hi Matthias,
>
> Gotcha, OK, the reason I thought that setting the number of ackers to
> zero would turn off acking for any bolts downstream from the KafkaSpout
> is the following guidance
> on http://storm.apache.org/documentation/Guaranteeing-message-processing.html:
>
> There are three ways to remove reliability. *The first is to set
> Config.TOPOLOGY_ACKERS to 0. In this case, Storm will call
> the |ack| method on the spout immediately after the spout emits a tuple.
> The tuple tree won't be tracked.*
>
> The second way is to remove reliability on a message by message basis.
> You can turn off tracking for an individual spout tuple by omitting a
> message id in the |SpoutOutputCollector.emit| method.
>
> Finally, if you don't care if a particular subset of the tuples
> downstream in the topology fail to be processed, you can emit them as
> unanchored tuples. Since they're not anchored to any spout tuples, they
> won't cause any spout tuples to fail if they aren't acked.
>
> --John
>
>
>
>
> On Wed, Dec 23, 2015 at 5:01 PM, Matthias J. Sax <mjsax@apache.org
> <ma...@apache.org>> wrote:
>
> Hi,
>
> if you want to disable fault-tolerance, you need to emit tuples in your
> spout without message IDs
>
> -> collector.emit(new Values(...)); // no messageId provided
>
> The parameter "topology.acker.executors" sets the number of ackers you
> want to use in your topology. I am not sure, if setting it to zero
> really disables fault-tolerance. (I would assume that fault-tolerance is
> still enabled and you get many failing tuples---due to timeout---because
> there are no ackers that can process the incoming ack messages.
>
> In any case, if your Bolts call "collector.ack(...)" this will be
> monitored in the UI. However, it does not necessarily mean that
> fault-tolerance is enabled or disabled. If you don't want to see an ack
> count in the UI, you need to remove the line of code in your
> Bolt.execute() methods that do the actual "collector.ack(...)" call.
>
> If you see a value for "acked" and "Complete latency" in your spouts,
> fault-tolerance is enabled. If both values stay zero (as well as
> "Failed") fault-tolerance if disabled.
>
> -Matthias
>
>
> On 12/23/2015 03:31 PM, John Yost wrote:
> > Hi Everyone,
> >
> > I have a topology that consists of a KafkaSpout and three
> downstream bolts.
> >
> > I turned off acking by setting topology.acker.executors to 0. It's my
> > understanding that I should only have acking in the KafkaSpout.
> However,
> > I see acking in all of my downstream bolts as well. Any ideas as
> to why?
> >
> > Thanks
> >
> > --John
>
>
Re: topology.ackers set to 0, still acking in downstream bolts?
Posted by John Yost <ho...@gmail.com>.
Hi Matthias,
Gotcha, OK, the reason I thought that setting the number of ackers to zero
would turn off acking for any bolts downstream from the KafkaSpout is the
following guidance on
http://storm.apache.org/documentation/Guaranteeing-message-processing.html:
There are three ways to remove reliability. *The first is to set
Config.TOPOLOGY_ACKERS to 0. In this case, Storm will call the ack method
on the spout immediately after the spout emits a tuple. The tuple tree
won't be tracked.*
The second way is to remove reliability on a message by message basis. You
can turn off tracking for an individual spout tuple by omitting a message
id in the SpoutOutputCollector.emit method.
Finally, if you don't care if a particular subset of the tuples downstream
in the topology fail to be processed, you can emit them as unanchored
tuples. Since they're not anchored to any spout tuples, they won't cause
any spout tuples to fail if they aren't acked.
--John
On Wed, Dec 23, 2015 at 5:01 PM, Matthias J. Sax <mj...@apache.org> wrote:
> Hi,
>
> if you want to disable fault-tolerance, you need to emit tuples in your
> spout without message IDs
>
> -> collector.emit(new Values(...)); // no messageId provided
>
> The parameter "topology.acker.executors" sets the number of ackers you
> want to use in your topology. I am not sure, if setting it to zero
> really disables fault-tolerance. (I would assume that fault-tolerance is
> still enabled and you get many failing tuples---due to timeout---because
> there are no ackers that can process the incoming ack messages.
>
> In any case, if your Bolts call "collector.ack(...)" this will be
> monitored in the UI. However, it does not necessarily mean that
> fault-tolerance is enabled or disabled. If you don't want to see an ack
> count in the UI, you need to remove the line of code in your
> Bolt.execute() methods that do the actual "collector.ack(...)" call.
>
> If you see a value for "acked" and "Complete latency" in your spouts,
> fault-tolerance is enabled. If both values stay zero (as well as
> "Failed") fault-tolerance if disabled.
>
> -Matthias
>
>
> On 12/23/2015 03:31 PM, John Yost wrote:
> > Hi Everyone,
> >
> > I have a topology that consists of a KafkaSpout and three downstream
> bolts.
> >
> > I turned off acking by setting topology.acker.executors to 0. It's my
> > understanding that I should only have acking in the KafkaSpout. However,
> > I see acking in all of my downstream bolts as well. Any ideas as to why?
> >
> > Thanks
> >
> > --John
>
>
Re: topology.ackers set to 0, still acking in downstream bolts?
Posted by "Matthias J. Sax" <mj...@apache.org>.
Hi,
if you want to disable fault-tolerance, you need to emit tuples in your
spout without message IDs
-> collector.emit(new Values(...)); // no messageId provided
The parameter "topology.acker.executors" sets the number of ackers you
want to use in your topology. I am not sure, if setting it to zero
really disables fault-tolerance. (I would assume that fault-tolerance is
still enabled and you get many failing tuples---due to timeout---because
there are no ackers that can process the incoming ack messages.
In any case, if your Bolts call "collector.ack(...)" this will be
monitored in the UI. However, it does not necessarily mean that
fault-tolerance is enabled or disabled. If you don't want to see an ack
count in the UI, you need to remove the line of code in your
Bolt.execute() methods that do the actual "collector.ack(...)" call.
If you see a value for "acked" and "Complete latency" in your spouts,
fault-tolerance is enabled. If both values stay zero (as well as
"Failed") fault-tolerance if disabled.
-Matthias
On 12/23/2015 03:31 PM, John Yost wrote:
> Hi Everyone,
>
> I have a topology that consists of a KafkaSpout and three downstream bolts.
>
> I turned off acking by setting topology.acker.executors to 0. It's my
> understanding that I should only have acking in the KafkaSpout. However,
> I see acking in all of my downstream bolts as well. Any ideas as to why?
>
> Thanks
>
> --John