You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by clay teahouse <cl...@gmail.com> on 2014/11/17 19:49:31 UTC
acking and offsets
Hello All,
I am using kafka spout that comes with storm 0.9.3 (
https://github.com/apache/storm).
I am having several different bolts consuming the same tuples from the
spout (in the same topology). These bolts process the tuples and send the
output to different destinations. I have a couple of basic questions:
1) What happens if the destination for some of the bolts is not accessible?
I can have a mechanism in each bolt that returns without consuming any
tuple if the destination is not available. But what happens to the tuples
at hand and when is the next execute is invoked.
- Do I need to have the destination availability check in a loop?
- Do I need to keep track of the offset?
2) Is it possible to ack in the bolts.
thanks,
Clay
Re: acking and offsets
Posted by clay teahouse <cl...@gmail.com>.
Thanks Kobi, for the explanation.
-Clay
On Wed, Nov 19, 2014 at 12:43 AM, Kobi Salant <ko...@liveperson.com> wrote:
> Hi Clay,
>
> If bolt4 fails then the spout will retry from the start.
>
> I am assuming you that either you are emitting your tuples with anchoring
> the original tuple or using BaseBasicBolt which does it for you.
>
> Please see the following link for detailed explanation
>
> https://storm.apache.org/documentation/Guaranteeing-message-processing.html
>
> Kobi
>
> On Wed, Nov 19, 2014 at 6:05 AM, clay teahouse <cl...@gmail.com>
> wrote:
>
>> 1) If in case of failure, the spout will retry from the start, even for
>> the bolts that didn't ack, then what is the point of acking?
>> 2) What happens when there is a chain of bolt, e.g.
>> spout1 --> bolt1 --> bolt2 --> bolt 3 --> bolt4 --> destination
>>
>> What happens if bolt4 fails, for whatever reason to deliver the tuples to
>> the destination? Does the spout retries?
>>
>> On Tue, Nov 18, 2014 at 12:42 AM, Kobi Salant <ko...@liveperson.com>
>> wrote:
>>
>>> Hi,
>>>
>>> The acking retry mechanism is based on the original tuple which left the
>>> spout. If it fails due to timeout or failure the spout will retry it from
>>> the start, it will not retry only the bolts that were not acking.
>>>
>>> You can try to save the IDs in each bolt and ignore them if they
>>> reappear.
>>>
>>> Kobi
>>>
>>> On Mon, Nov 17, 2014 at 9:40 PM, clay teahouse <cl...@gmail.com>
>>> wrote:
>>>
>>>> Just to clarify my question regarding the ack:
>>>>
>>>> With the use case I mentioned, it should have said What is the default
>>>> behavior with regard to acking in bolts and how the concept of offset and
>>>> acking interact? Is it possible to not ack in a bolt to achieve the outcome
>>>> I am looking for.
>>>>
>>>> thanks.
>>>>
>>>> On Mon, Nov 17, 2014 at 12:49 PM, clay teahouse <clayteahouse@gmail.com
>>>> > wrote:
>>>>
>>>>> Hello All,
>>>>> I am using kafka spout that comes with storm 0.9.3 (
>>>>> https://github.com/apache/storm).
>>>>> I am having several different bolts consuming the same tuples from the
>>>>> spout (in the same topology). These bolts process the tuples and send the
>>>>> output to different destinations. I have a couple of basic questions:
>>>>> 1) What happens if the destination for some of the bolts is not
>>>>> accessible? I can have a mechanism in each bolt that returns without
>>>>> consuming any tuple if the destination is not available. But what happens
>>>>> to the tuples at hand and when is the next execute is invoked.
>>>>> - Do I need to have the destination availability check in a loop?
>>>>> - Do I need to keep track of the offset?
>>>>> 2) Is it possible to ack in the bolts.
>>>>>
>>>>> thanks,
>>>>> Clay
>>>>>
>>>>
>>>>
>>>
>>> This message may contain confidential and/or privileged information.
>>> If you are not the addressee or authorized to receive this on behalf of
>>> the addressee you must not use, copy, disclose or take action based on this
>>> message or any information herein.
>>> If you have received this message in error, please advise the sender
>>> immediately by reply email and delete this message. Thank you.
>>>
>>
>>
>
> This message may contain confidential and/or privileged information.
> If you are not the addressee or authorized to receive this on behalf of
> the addressee you must not use, copy, disclose or take action based on this
> message or any information herein.
> If you have received this message in error, please advise the sender
> immediately by reply email and delete this message. Thank you.
>
Re: acking and offsets
Posted by Kobi Salant <ko...@liveperson.com>.
Hi Clay,
If bolt4 fails then the spout will retry from the start.
I am assuming you that either you are emitting your tuples with anchoring
the original tuple or using BaseBasicBolt which does it for you.
Please see the following link for detailed explanation
https://storm.apache.org/documentation/Guaranteeing-message-processing.html
Kobi
On Wed, Nov 19, 2014 at 6:05 AM, clay teahouse <cl...@gmail.com>
wrote:
> 1) If in case of failure, the spout will retry from the start, even for
> the bolts that didn't ack, then what is the point of acking?
> 2) What happens when there is a chain of bolt, e.g.
> spout1 --> bolt1 --> bolt2 --> bolt 3 --> bolt4 --> destination
>
> What happens if bolt4 fails, for whatever reason to deliver the tuples to
> the destination? Does the spout retries?
>
> On Tue, Nov 18, 2014 at 12:42 AM, Kobi Salant <ko...@liveperson.com>
> wrote:
>
>> Hi,
>>
>> The acking retry mechanism is based on the original tuple which left the
>> spout. If it fails due to timeout or failure the spout will retry it from
>> the start, it will not retry only the bolts that were not acking.
>>
>> You can try to save the IDs in each bolt and ignore them if they reappear.
>>
>> Kobi
>>
>> On Mon, Nov 17, 2014 at 9:40 PM, clay teahouse <cl...@gmail.com>
>> wrote:
>>
>>> Just to clarify my question regarding the ack:
>>>
>>> With the use case I mentioned, it should have said What is the default
>>> behavior with regard to acking in bolts and how the concept of offset and
>>> acking interact? Is it possible to not ack in a bolt to achieve the outcome
>>> I am looking for.
>>>
>>> thanks.
>>>
>>> On Mon, Nov 17, 2014 at 12:49 PM, clay teahouse <cl...@gmail.com>
>>> wrote:
>>>
>>>> Hello All,
>>>> I am using kafka spout that comes with storm 0.9.3 (
>>>> https://github.com/apache/storm).
>>>> I am having several different bolts consuming the same tuples from the
>>>> spout (in the same topology). These bolts process the tuples and send the
>>>> output to different destinations. I have a couple of basic questions:
>>>> 1) What happens if the destination for some of the bolts is not
>>>> accessible? I can have a mechanism in each bolt that returns without
>>>> consuming any tuple if the destination is not available. But what happens
>>>> to the tuples at hand and when is the next execute is invoked.
>>>> - Do I need to have the destination availability check in a loop?
>>>> - Do I need to keep track of the offset?
>>>> 2) Is it possible to ack in the bolts.
>>>>
>>>> thanks,
>>>> Clay
>>>>
>>>
>>>
>>
>> This message may contain confidential and/or privileged information.
>> If you are not the addressee or authorized to receive this on behalf of
>> the addressee you must not use, copy, disclose or take action based on this
>> message or any information herein.
>> If you have received this message in error, please advise the sender
>> immediately by reply email and delete this message. Thank you.
>>
>
>
--
This message may contain confidential and/or privileged information.
If you are not the addressee or authorized to receive this on behalf of the
addressee you must not use, copy, disclose or take action based on this
message or any information herein.
If you have received this message in error, please advise the sender
immediately by reply email and delete this message. Thank you.
Re: acking and offsets
Posted by clay teahouse <cl...@gmail.com>.
1) If in case of failure, the spout will retry from the start, even for the
bolts that didn't ack, then what is the point of acking?
2) What happens when there is a chain of bolt, e.g.
spout1 --> bolt1 --> bolt2 --> bolt 3 --> bolt4 --> destination
What happens if bolt4 fails, for whatever reason to deliver the tuples to
the destination? Does the spout retries?
On Tue, Nov 18, 2014 at 12:42 AM, Kobi Salant <ko...@liveperson.com> wrote:
> Hi,
>
> The acking retry mechanism is based on the original tuple which left the
> spout. If it fails due to timeout or failure the spout will retry it from
> the start, it will not retry only the bolts that were not acking.
>
> You can try to save the IDs in each bolt and ignore them if they reappear.
>
> Kobi
>
> On Mon, Nov 17, 2014 at 9:40 PM, clay teahouse <cl...@gmail.com>
> wrote:
>
>> Just to clarify my question regarding the ack:
>>
>> With the use case I mentioned, it should have said What is the default
>> behavior with regard to acking in bolts and how the concept of offset and
>> acking interact? Is it possible to not ack in a bolt to achieve the outcome
>> I am looking for.
>>
>> thanks.
>>
>> On Mon, Nov 17, 2014 at 12:49 PM, clay teahouse <cl...@gmail.com>
>> wrote:
>>
>>> Hello All,
>>> I am using kafka spout that comes with storm 0.9.3 (
>>> https://github.com/apache/storm).
>>> I am having several different bolts consuming the same tuples from the
>>> spout (in the same topology). These bolts process the tuples and send the
>>> output to different destinations. I have a couple of basic questions:
>>> 1) What happens if the destination for some of the bolts is not
>>> accessible? I can have a mechanism in each bolt that returns without
>>> consuming any tuple if the destination is not available. But what happens
>>> to the tuples at hand and when is the next execute is invoked.
>>> - Do I need to have the destination availability check in a loop?
>>> - Do I need to keep track of the offset?
>>> 2) Is it possible to ack in the bolts.
>>>
>>> thanks,
>>> Clay
>>>
>>
>>
>
> This message may contain confidential and/or privileged information.
> If you are not the addressee or authorized to receive this on behalf of
> the addressee you must not use, copy, disclose or take action based on this
> message or any information herein.
> If you have received this message in error, please advise the sender
> immediately by reply email and delete this message. Thank you.
>
Re: acking and offsets
Posted by clay teahouse <cl...@gmail.com>.
1) If in case of failure, the spout will retry from the start, even for the
bolts that didn't ack, then what is the point of acking?
2) What happens when there is a chain of bolt, e.g.
spout1 --> bolt1 --> bolt2 --> bolt 3 --> bolt4 --> destination
What happens if bolt4 fails, for whatever reason to deliver the tuples to
the destination? Does the spout retries?
On Tue, Nov 18, 2014 at 12:42 AM, Kobi Salant <ko...@liveperson.com> wrote:
> Hi,
>
> The acking retry mechanism is based on the original tuple which left the
> spout. If it fails due to timeout or failure the spout will retry it from
> the start, it will not retry only the bolts that were not acking.
>
> You can try to save the IDs in each bolt and ignore them if they reappear.
>
> Kobi
>
> On Mon, Nov 17, 2014 at 9:40 PM, clay teahouse <cl...@gmail.com>
> wrote:
>
>> Just to clarify my question regarding the ack:
>>
>> With the use case I mentioned, it should have said What is the default
>> behavior with regard to acking in bolts and how the concept of offset and
>> acking interact? Is it possible to not ack in a bolt to achieve the outcome
>> I am looking for.
>>
>> thanks.
>>
>> On Mon, Nov 17, 2014 at 12:49 PM, clay teahouse <cl...@gmail.com>
>> wrote:
>>
>>> Hello All,
>>> I am using kafka spout that comes with storm 0.9.3 (
>>> https://github.com/apache/storm).
>>> I am having several different bolts consuming the same tuples from the
>>> spout (in the same topology). These bolts process the tuples and send the
>>> output to different destinations. I have a couple of basic questions:
>>> 1) What happens if the destination for some of the bolts is not
>>> accessible? I can have a mechanism in each bolt that returns without
>>> consuming any tuple if the destination is not available. But what happens
>>> to the tuples at hand and when is the next execute is invoked.
>>> - Do I need to have the destination availability check in a loop?
>>> - Do I need to keep track of the offset?
>>> 2) Is it possible to ack in the bolts.
>>>
>>> thanks,
>>> Clay
>>>
>>
>>
>
> This message may contain confidential and/or privileged information.
> If you are not the addressee or authorized to receive this on behalf of
> the addressee you must not use, copy, disclose or take action based on this
> message or any information herein.
> If you have received this message in error, please advise the sender
> immediately by reply email and delete this message. Thank you.
>
Re: acking and offsets
Posted by Kobi Salant <ko...@liveperson.com>.
Hi,
The acking retry mechanism is based on the original tuple which left the
spout. If it fails due to timeout or failure the spout will retry it from
the start, it will not retry only the bolts that were not acking.
You can try to save the IDs in each bolt and ignore them if they reappear.
Kobi
On Mon, Nov 17, 2014 at 9:40 PM, clay teahouse <cl...@gmail.com>
wrote:
> Just to clarify my question regarding the ack:
>
> With the use case I mentioned, it should have said What is the default
> behavior with regard to acking in bolts and how the concept of offset and
> acking interact? Is it possible to not ack in a bolt to achieve the outcome
> I am looking for.
>
> thanks.
>
> On Mon, Nov 17, 2014 at 12:49 PM, clay teahouse <cl...@gmail.com>
> wrote:
>
>> Hello All,
>> I am using kafka spout that comes with storm 0.9.3 (
>> https://github.com/apache/storm).
>> I am having several different bolts consuming the same tuples from the
>> spout (in the same topology). These bolts process the tuples and send the
>> output to different destinations. I have a couple of basic questions:
>> 1) What happens if the destination for some of the bolts is not
>> accessible? I can have a mechanism in each bolt that returns without
>> consuming any tuple if the destination is not available. But what happens
>> to the tuples at hand and when is the next execute is invoked.
>> - Do I need to have the destination availability check in a loop?
>> - Do I need to keep track of the offset?
>> 2) Is it possible to ack in the bolts.
>>
>> thanks,
>> Clay
>>
>
>
--
This message may contain confidential and/or privileged information.
If you are not the addressee or authorized to receive this on behalf of the
addressee you must not use, copy, disclose or take action based on this
message or any information herein.
If you have received this message in error, please advise the sender
immediately by reply email and delete this message. Thank you.
Re: acking and offsets
Posted by clay teahouse <cl...@gmail.com>.
Just to clarify my question regarding the ack:
With the use case I mentioned, it should have said What is the default
behavior with regard to acking in bolts and how the concept of offset and
acking interact? Is it possible to not ack in a bolt to achieve the outcome
I am looking for.
thanks.
On Mon, Nov 17, 2014 at 12:49 PM, clay teahouse <cl...@gmail.com>
wrote:
> Hello All,
> I am using kafka spout that comes with storm 0.9.3 (
> https://github.com/apache/storm).
> I am having several different bolts consuming the same tuples from the
> spout (in the same topology). These bolts process the tuples and send the
> output to different destinations. I have a couple of basic questions:
> 1) What happens if the destination for some of the bolts is not
> accessible? I can have a mechanism in each bolt that returns without
> consuming any tuple if the destination is not available. But what happens
> to the tuples at hand and when is the next execute is invoked.
> - Do I need to have the destination availability check in a loop?
> - Do I need to keep track of the offset?
> 2) Is it possible to ack in the bolts.
>
> thanks,
> Clay
>
Re: acking and offsets
Posted by clay teahouse <cl...@gmail.com>.
Just to clarify my question regarding the ack:
With the use case I mentioned, it should have said What is the default
behavior with regard to acking in bolts and how the concept of offset and
acking interact? Is it possible to not ack in a bolt to achieve the outcome
I am looking for.
thanks.
On Mon, Nov 17, 2014 at 12:49 PM, clay teahouse <cl...@gmail.com>
wrote:
> Hello All,
> I am using kafka spout that comes with storm 0.9.3 (
> https://github.com/apache/storm).
> I am having several different bolts consuming the same tuples from the
> spout (in the same topology). These bolts process the tuples and send the
> output to different destinations. I have a couple of basic questions:
> 1) What happens if the destination for some of the bolts is not
> accessible? I can have a mechanism in each bolt that returns without
> consuming any tuple if the destination is not available. But what happens
> to the tuples at hand and when is the next execute is invoked.
> - Do I need to have the destination availability check in a loop?
> - Do I need to keep track of the offset?
> 2) Is it possible to ack in the bolts.
>
> thanks,
> Clay
>