You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Kushan Maskey <ku...@mmillerassociates.com> on 2014/08/26 00:42:18 UTC
Question on failing ack
I have set up topology to load a very large volume of data. Recently I just
loaded about 60K records and found out that there are some failed acks on
few spouts but non on the bolts. Storm completed running and seem to look
stable. Initially i started with a lesser amount of data like about 500
records successfully and then increased up to 60K where i saw the failed
acks.
Questions:
1. Does that mean that the spout was not able to read some messages from
Kafka? Since there are no failed ack on the bolts as per UI, what ever the
message received has been successfully processed by the bolts.
2. how do i interpret the numbers of failed acks like this acked:315500
and failed: 2980.
Does this mean that 2980 records failed to be processed? Is this is the
case then, how do I avoid this from happening because I will be loosing
2980 records.
3. I also see that few of the records failed to be inserted into Cassandra
database. What is the best way to reprocess the data again as it is quite
difficult to do it through the batch process that I am currently running.
LMK, thanks.
--
Kushan Maskey
817.403.7500
Re: Question on failing ack
Posted by Kushan Maskey <ku...@mmillerassociates.com>.
Thanks Srinath for your response.
I have not looked into configuring the buffers. If you can point me to the
direction where I can get more information on it would be helpful. AFAIK
the records from bulk load is mostly at constant rate. When I am just doing
a message at a time it works fine. Also the failed ack number 2980, does
that really mean that many records failed to ack?
--
Kushan Maskey
817.403.7500
On Mon, Aug 25, 2014 at 7:46 PM, Srinath C <sr...@gmail.com> wrote:
> I would suspect that at some point the rate at which the spouts emitted
> exceeded the rate at which the bolts could process. Maybe you could look at
> configuring the buffers (if you haven't yet done that). Do your records get
> processed at a constant rate?
>
>
> On Tue, Aug 26, 2014 at 4:12 AM, Kushan Maskey <
> kushan.maskey@mmillerassociates.com> wrote:
>
>> I have set up topology to load a very large volume of data. Recently I
>> just loaded about 60K records and found out that there are some failed acks
>> on few spouts but non on the bolts. Storm completed running and seem to
>> look stable. Initially i started with a lesser amount of data like about
>> 500 records successfully and then increased up to 60K where i saw the
>> failed acks.
>>
>> Questions:
>> 1. Does that mean that the spout was not able to read some messages from
>> Kafka? Since there are no failed ack on the bolts as per UI, what ever the
>> message received has been successfully processed by the bolts.
>> 2. how do i interpret the numbers of failed acks like this acked:315500
>> and failed: 2980.
>> Does this mean that 2980 records failed to be processed? Is this is the
>> case then, how do I avoid this from happening because I will be loosing
>> 2980 records.
>> 3. I also see that few of the records failed to be inserted into
>> Cassandra database. What is the best way to reprocess the data again as it
>> is quite difficult to do it through the batch process that I am currently
>> running.
>>
>> LMK, thanks.
>>
>> --
>> Kushan Maskey
>> 817.403.7500
>>
>
>
Re: Question on failing ack
Posted by Srinath C <sr...@gmail.com>.
I would suspect that at some point the rate at which the spouts emitted
exceeded the rate at which the bolts could process. Maybe you could look at
configuring the buffers (if you haven't yet done that). Do your records get
processed at a constant rate?
On Tue, Aug 26, 2014 at 4:12 AM, Kushan Maskey <
kushan.maskey@mmillerassociates.com> wrote:
> I have set up topology to load a very large volume of data. Recently I
> just loaded about 60K records and found out that there are some failed acks
> on few spouts but non on the bolts. Storm completed running and seem to
> look stable. Initially i started with a lesser amount of data like about
> 500 records successfully and then increased up to 60K where i saw the
> failed acks.
>
> Questions:
> 1. Does that mean that the spout was not able to read some messages from
> Kafka? Since there are no failed ack on the bolts as per UI, what ever the
> message received has been successfully processed by the bolts.
> 2. how do i interpret the numbers of failed acks like this acked:315500
> and failed: 2980.
> Does this mean that 2980 records failed to be processed? Is this is the
> case then, how do I avoid this from happening because I will be loosing
> 2980 records.
> 3. I also see that few of the records failed to be inserted into Cassandra
> database. What is the best way to reprocess the data again as it is quite
> difficult to do it through the batch process that I am currently running.
>
> LMK, thanks.
>
> --
> Kushan Maskey
> 817.403.7500
>
Re: Question on failing ack
Posted by Srinath C <sr...@gmail.com>.
Yes, the defaults mentioned in the blog worked for me too.
On Wed, Aug 27, 2014 at 2:49 AM, Kushan Maskey <
kushan.maskey@mmillerassociates.com> wrote:
> These changes did help a lot. Now I dont see any failed acks for alsmot
> 70k data load. Thanks a lot for your help.
>
> --
> Kushan Maskey
> 817.403.7500
>
>
> On Tue, Aug 26, 2014 at 9:45 AM, Kushan Maskey <
> kushan.maskey@mmillerassociates.com> wrote:
>
>> Also FYI, I do not have any failure in any of my bolts. So I am guessing
>> it has to do with the amount of message that the spout is trying to read
>> form kafka.
>>
>> as per the document by Michael Noll I am trying to see if setting up the
>> config as per suggestion as
>>
>> config.put(Config.TOPOLOGY_RECEIVER_BUFFER_SIZE, 8);
>>
>> config.put(Config.TOPOLOGY_TRANSFER_BUFFER_SIZE, 32);
>>
>> config.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE, 16384);
>>
>> config.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE, 16384);
>>
>>
>> Let me know if any of you have any suggestion. Thanks.
>>
>> --
>> Kushan Maskey
>>
>>
>>
>> On Tue, Aug 26, 2014 at 9:28 AM, Kushan Maskey <
>> kushan.maskey@mmillerassociates.com> wrote:
>>
>>> I started looking into setting up internal message buffer as mentioned
>>> in this link.
>>>
>>> http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers/#how-to-configure-storms-internal-message-buffers
>>>
>>> I found out that my message size could be as big as 10K. So does that
>>> mean that I should set the buffer size about 10K?
>>>
>>> --
>>> Kushan Maskey
>>> 817.403.7500
>>>
>>>
>>> On Tue, Aug 26, 2014 at 7:45 AM, Kushan Maskey <
>>> kushan.maskey@mmillerassociates.com> wrote:
>>>
>>>> Thanks, Michael,
>>>>
>>>> How do you verify the reliability of the KafkaSpout? I am using the
>>>> KafkaSpout that came with storm 0.9.2. AFAIK kafkaSpout is quite reliable.
>>>> I am guessing it the processing time for each record in the bolt. Yes form
>>>> the log I do see few Cassandra exceptions while inserting the records.
>>>>
>>>> --
>>>> Kushan Maskey
>>>> 817.403.7500
>>>>
>>>>
>>>> On Mon, Aug 25, 2014 at 9:39 PM, Michael Rose <mi...@fullcontact.com>
>>>> wrote:
>>>>
>>>>> Hi Kushan,
>>>>>
>>>>> Depending on the Kafka spout you're using, it could be doing different
>>>>> things when it failed. However, if it's running reliably, the Cassandra
>>>>> insertion failures would have forced a replay from the spout until they had
>>>>> completed.
>>>>>
>>>>> Michael Rose (@Xorlev <https://twitter.com/xorlev>)
>>>>> Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
>>>>> michael@fullcontact.com
>>>>>
>>>>>
>>>>> On Mon, Aug 25, 2014 at 4:42 PM, Kushan Maskey <
>>>>> kushan.maskey@mmillerassociates.com> wrote:
>>>>>
>>>>>> I have set up topology to load a very large volume of data. Recently
>>>>>> I just loaded about 60K records and found out that there are some failed
>>>>>> acks on few spouts but non on the bolts. Storm completed running and seem
>>>>>> to look stable. Initially i started with a lesser amount of data like about
>>>>>> 500 records successfully and then increased up to 60K where i saw the
>>>>>> failed acks.
>>>>>>
>>>>>> Questions:
>>>>>> 1. Does that mean that the spout was not able to read some messages
>>>>>> from Kafka? Since there are no failed ack on the bolts as per UI, what ever
>>>>>> the message received has been successfully processed by the bolts.
>>>>>> 2. how do i interpret the numbers of failed acks like this acked:315500
>>>>>> and failed: 2980.
>>>>>> Does this mean that 2980 records failed to be processed? Is this is
>>>>>> the case then, how do I avoid this from happening because I will be loosing
>>>>>> 2980 records.
>>>>>> 3. I also see that few of the records failed to be inserted into
>>>>>> Cassandra database. What is the best way to reprocess the data again as it
>>>>>> is quite difficult to do it through the batch process that I am currently
>>>>>> running.
>>>>>>
>>>>>> LMK, thanks.
>>>>>>
>>>>>> --
>>>>>> Kushan Maskey
>>>>>> 817.403.7500
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Question on failing ack
Posted by Kushan Maskey <ku...@mmillerassociates.com>.
These changes did help a lot. Now I dont see any failed acks for alsmot 70k
data load. Thanks a lot for your help.
--
Kushan Maskey
817.403.7500
On Tue, Aug 26, 2014 at 9:45 AM, Kushan Maskey <
kushan.maskey@mmillerassociates.com> wrote:
> Also FYI, I do not have any failure in any of my bolts. So I am guessing
> it has to do with the amount of message that the spout is trying to read
> form kafka.
>
> as per the document by Michael Noll I am trying to see if setting up the
> config as per suggestion as
>
> config.put(Config.TOPOLOGY_RECEIVER_BUFFER_SIZE, 8);
>
> config.put(Config.TOPOLOGY_TRANSFER_BUFFER_SIZE, 32);
>
> config.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE, 16384);
>
> config.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE, 16384);
>
>
> Let me know if any of you have any suggestion. Thanks.
>
> --
> Kushan Maskey
>
>
>
> On Tue, Aug 26, 2014 at 9:28 AM, Kushan Maskey <
> kushan.maskey@mmillerassociates.com> wrote:
>
>> I started looking into setting up internal message buffer as mentioned in
>> this link.
>>
>> http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers/#how-to-configure-storms-internal-message-buffers
>>
>> I found out that my message size could be as big as 10K. So does that
>> mean that I should set the buffer size about 10K?
>>
>> --
>> Kushan Maskey
>> 817.403.7500
>>
>>
>> On Tue, Aug 26, 2014 at 7:45 AM, Kushan Maskey <
>> kushan.maskey@mmillerassociates.com> wrote:
>>
>>> Thanks, Michael,
>>>
>>> How do you verify the reliability of the KafkaSpout? I am using the
>>> KafkaSpout that came with storm 0.9.2. AFAIK kafkaSpout is quite reliable.
>>> I am guessing it the processing time for each record in the bolt. Yes form
>>> the log I do see few Cassandra exceptions while inserting the records.
>>>
>>> --
>>> Kushan Maskey
>>> 817.403.7500
>>>
>>>
>>> On Mon, Aug 25, 2014 at 9:39 PM, Michael Rose <mi...@fullcontact.com>
>>> wrote:
>>>
>>>> Hi Kushan,
>>>>
>>>> Depending on the Kafka spout you're using, it could be doing different
>>>> things when it failed. However, if it's running reliably, the Cassandra
>>>> insertion failures would have forced a replay from the spout until they had
>>>> completed.
>>>>
>>>> Michael Rose (@Xorlev <https://twitter.com/xorlev>)
>>>> Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
>>>> michael@fullcontact.com
>>>>
>>>>
>>>> On Mon, Aug 25, 2014 at 4:42 PM, Kushan Maskey <
>>>> kushan.maskey@mmillerassociates.com> wrote:
>>>>
>>>>> I have set up topology to load a very large volume of data. Recently I
>>>>> just loaded about 60K records and found out that there are some failed acks
>>>>> on few spouts but non on the bolts. Storm completed running and seem to
>>>>> look stable. Initially i started with a lesser amount of data like about
>>>>> 500 records successfully and then increased up to 60K where i saw the
>>>>> failed acks.
>>>>>
>>>>> Questions:
>>>>> 1. Does that mean that the spout was not able to read some messages
>>>>> from Kafka? Since there are no failed ack on the bolts as per UI, what ever
>>>>> the message received has been successfully processed by the bolts.
>>>>> 2. how do i interpret the numbers of failed acks like this acked:315500
>>>>> and failed: 2980.
>>>>> Does this mean that 2980 records failed to be processed? Is this is
>>>>> the case then, how do I avoid this from happening because I will be loosing
>>>>> 2980 records.
>>>>> 3. I also see that few of the records failed to be inserted into
>>>>> Cassandra database. What is the best way to reprocess the data again as it
>>>>> is quite difficult to do it through the batch process that I am currently
>>>>> running.
>>>>>
>>>>> LMK, thanks.
>>>>>
>>>>> --
>>>>> Kushan Maskey
>>>>> 817.403.7500
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Question on failing ack
Posted by Kushan Maskey <ku...@mmillerassociates.com>.
Also FYI, I do not have any failure in any of my bolts. So I am guessing it
has to do with the amount of message that the spout is trying to read form
kafka.
as per the document by Michael Noll I am trying to see if setting up the
config as per suggestion as
config.put(Config.TOPOLOGY_RECEIVER_BUFFER_SIZE, 8);
config.put(Config.TOPOLOGY_TRANSFER_BUFFER_SIZE, 32);
config.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE, 16384);
config.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE, 16384);
Let me know if any of you have any suggestion. Thanks.
--
Kushan Maskey
On Tue, Aug 26, 2014 at 9:28 AM, Kushan Maskey <
kushan.maskey@mmillerassociates.com> wrote:
> I started looking into setting up internal message buffer as mentioned in
> this link.
>
> http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers/#how-to-configure-storms-internal-message-buffers
>
> I found out that my message size could be as big as 10K. So does that mean
> that I should set the buffer size about 10K?
>
> --
> Kushan Maskey
> 817.403.7500
>
>
> On Tue, Aug 26, 2014 at 7:45 AM, Kushan Maskey <
> kushan.maskey@mmillerassociates.com> wrote:
>
>> Thanks, Michael,
>>
>> How do you verify the reliability of the KafkaSpout? I am using the
>> KafkaSpout that came with storm 0.9.2. AFAIK kafkaSpout is quite reliable.
>> I am guessing it the processing time for each record in the bolt. Yes form
>> the log I do see few Cassandra exceptions while inserting the records.
>>
>> --
>> Kushan Maskey
>> 817.403.7500
>>
>>
>> On Mon, Aug 25, 2014 at 9:39 PM, Michael Rose <mi...@fullcontact.com>
>> wrote:
>>
>>> Hi Kushan,
>>>
>>> Depending on the Kafka spout you're using, it could be doing different
>>> things when it failed. However, if it's running reliably, the Cassandra
>>> insertion failures would have forced a replay from the spout until they had
>>> completed.
>>>
>>> Michael Rose (@Xorlev <https://twitter.com/xorlev>)
>>> Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
>>> michael@fullcontact.com
>>>
>>>
>>> On Mon, Aug 25, 2014 at 4:42 PM, Kushan Maskey <
>>> kushan.maskey@mmillerassociates.com> wrote:
>>>
>>>> I have set up topology to load a very large volume of data. Recently I
>>>> just loaded about 60K records and found out that there are some failed acks
>>>> on few spouts but non on the bolts. Storm completed running and seem to
>>>> look stable. Initially i started with a lesser amount of data like about
>>>> 500 records successfully and then increased up to 60K where i saw the
>>>> failed acks.
>>>>
>>>> Questions:
>>>> 1. Does that mean that the spout was not able to read some messages
>>>> from Kafka? Since there are no failed ack on the bolts as per UI, what ever
>>>> the message received has been successfully processed by the bolts.
>>>> 2. how do i interpret the numbers of failed acks like this acked:315500
>>>> and failed: 2980.
>>>> Does this mean that 2980 records failed to be processed? Is this is the
>>>> case then, how do I avoid this from happening because I will be loosing
>>>> 2980 records.
>>>> 3. I also see that few of the records failed to be inserted into
>>>> Cassandra database. What is the best way to reprocess the data again as it
>>>> is quite difficult to do it through the batch process that I am currently
>>>> running.
>>>>
>>>> LMK, thanks.
>>>>
>>>> --
>>>> Kushan Maskey
>>>> 817.403.7500
>>>>
>>>
>>>
>>
>
Re: Question on failing ack
Posted by Kushan Maskey <ku...@mmillerassociates.com>.
I started looking into setting up internal message buffer as mentioned in
this link.
http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers/#how-to-configure-storms-internal-message-buffers
I found out that my message size could be as big as 10K. So does that mean
that I should set the buffer size about 10K?
--
Kushan Maskey
817.403.7500
On Tue, Aug 26, 2014 at 7:45 AM, Kushan Maskey <
kushan.maskey@mmillerassociates.com> wrote:
> Thanks, Michael,
>
> How do you verify the reliability of the KafkaSpout? I am using the
> KafkaSpout that came with storm 0.9.2. AFAIK kafkaSpout is quite reliable.
> I am guessing it the processing time for each record in the bolt. Yes form
> the log I do see few Cassandra exceptions while inserting the records.
>
> --
> Kushan Maskey
> 817.403.7500
>
>
> On Mon, Aug 25, 2014 at 9:39 PM, Michael Rose <mi...@fullcontact.com>
> wrote:
>
>> Hi Kushan,
>>
>> Depending on the Kafka spout you're using, it could be doing different
>> things when it failed. However, if it's running reliably, the Cassandra
>> insertion failures would have forced a replay from the spout until they had
>> completed.
>>
>> Michael Rose (@Xorlev <https://twitter.com/xorlev>)
>> Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
>> michael@fullcontact.com
>>
>>
>> On Mon, Aug 25, 2014 at 4:42 PM, Kushan Maskey <
>> kushan.maskey@mmillerassociates.com> wrote:
>>
>>> I have set up topology to load a very large volume of data. Recently I
>>> just loaded about 60K records and found out that there are some failed acks
>>> on few spouts but non on the bolts. Storm completed running and seem to
>>> look stable. Initially i started with a lesser amount of data like about
>>> 500 records successfully and then increased up to 60K where i saw the
>>> failed acks.
>>>
>>> Questions:
>>> 1. Does that mean that the spout was not able to read some messages from
>>> Kafka? Since there are no failed ack on the bolts as per UI, what ever the
>>> message received has been successfully processed by the bolts.
>>> 2. how do i interpret the numbers of failed acks like this acked:315500
>>> and failed: 2980.
>>> Does this mean that 2980 records failed to be processed? Is this is the
>>> case then, how do I avoid this from happening because I will be loosing
>>> 2980 records.
>>> 3. I also see that few of the records failed to be inserted into
>>> Cassandra database. What is the best way to reprocess the data again as it
>>> is quite difficult to do it through the batch process that I am currently
>>> running.
>>>
>>> LMK, thanks.
>>>
>>> --
>>> Kushan Maskey
>>> 817.403.7500
>>>
>>
>>
>
Re: Question on failing ack
Posted by Kushan Maskey <ku...@mmillerassociates.com>.
Thanks, Michael,
How do you verify the reliability of the KafkaSpout? I am using the
KafkaSpout that came with storm 0.9.2. AFAIK kafkaSpout is quite reliable.
I am guessing it the processing time for each record in the bolt. Yes form
the log I do see few Cassandra exceptions while inserting the records.
--
Kushan Maskey
817.403.7500
On Mon, Aug 25, 2014 at 9:39 PM, Michael Rose <mi...@fullcontact.com>
wrote:
> Hi Kushan,
>
> Depending on the Kafka spout you're using, it could be doing different
> things when it failed. However, if it's running reliably, the Cassandra
> insertion failures would have forced a replay from the spout until they had
> completed.
>
> Michael Rose (@Xorlev <https://twitter.com/xorlev>)
> Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
> michael@fullcontact.com
>
>
> On Mon, Aug 25, 2014 at 4:42 PM, Kushan Maskey <
> kushan.maskey@mmillerassociates.com> wrote:
>
>> I have set up topology to load a very large volume of data. Recently I
>> just loaded about 60K records and found out that there are some failed acks
>> on few spouts but non on the bolts. Storm completed running and seem to
>> look stable. Initially i started with a lesser amount of data like about
>> 500 records successfully and then increased up to 60K where i saw the
>> failed acks.
>>
>> Questions:
>> 1. Does that mean that the spout was not able to read some messages from
>> Kafka? Since there are no failed ack on the bolts as per UI, what ever the
>> message received has been successfully processed by the bolts.
>> 2. how do i interpret the numbers of failed acks like this acked:315500
>> and failed: 2980.
>> Does this mean that 2980 records failed to be processed? Is this is the
>> case then, how do I avoid this from happening because I will be loosing
>> 2980 records.
>> 3. I also see that few of the records failed to be inserted into
>> Cassandra database. What is the best way to reprocess the data again as it
>> is quite difficult to do it through the batch process that I am currently
>> running.
>>
>> LMK, thanks.
>>
>> --
>> Kushan Maskey
>> 817.403.7500
>>
>
>
Re: Question on failing ack
Posted by Michael Rose <mi...@fullcontact.com>.
Hi Kushan,
Depending on the Kafka spout you're using, it could be doing different
things when it failed. However, if it's running reliably, the Cassandra
insertion failures would have forced a replay from the spout until they had
completed.
Michael Rose (@Xorlev <https://twitter.com/xorlev>)
Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
michael@fullcontact.com
On Mon, Aug 25, 2014 at 4:42 PM, Kushan Maskey <
kushan.maskey@mmillerassociates.com> wrote:
> I have set up topology to load a very large volume of data. Recently I
> just loaded about 60K records and found out that there are some failed acks
> on few spouts but non on the bolts. Storm completed running and seem to
> look stable. Initially i started with a lesser amount of data like about
> 500 records successfully and then increased up to 60K where i saw the
> failed acks.
>
> Questions:
> 1. Does that mean that the spout was not able to read some messages from
> Kafka? Since there are no failed ack on the bolts as per UI, what ever the
> message received has been successfully processed by the bolts.
> 2. how do i interpret the numbers of failed acks like this acked:315500
> and failed: 2980.
> Does this mean that 2980 records failed to be processed? Is this is the
> case then, how do I avoid this from happening because I will be loosing
> 2980 records.
> 3. I also see that few of the records failed to be inserted into Cassandra
> database. What is the best way to reprocess the data again as it is quite
> difficult to do it through the batch process that I am currently running.
>
> LMK, thanks.
>
> --
> Kushan Maskey
> 817.403.7500
>