You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Chitra Raveendran <ch...@flutura.com> on 2014/02/10 09:12:28 UTC
Deactivate topology incase of network issues!!!
Hi
I have a Realtime data crunching pipeline which wherein storm consumes data
from kafka, we do some processing on this data (basically counts and
aggregations) and store the results into a MySql database.( I use the
default Kafka spout.)
Over the weekend some network related issues led to MySQL server crash, as
a result storm kept re-sending the messages, the processing in the storm
supervisors increased drastically and CPU usage hit 100%. This in turn led
to some supervisors falling out of the cluster and trying to restart.
How would I be able to *handle* such an unforeseen situation? Is there any
way through which storm would understand that there is a MySql server issue
and stop re-sending data?
Is there anyway that this exception can be caught and in *topology be
deactivated* for a while?
Regards,
Chitra
Re: Deactivate topology incase of network issues!!!
Posted by Chitra Raveendran <ch...@flutura.com>.
Thank you Enno !!
I'm going to test out with 5K and see check the performance! Thanks again
On Mon, Feb 10, 2014 at 4:32 PM, Enno Shioji <en...@peerindex.com>wrote:
> > What would be an appropriate value for TOPOLOGY_MAX_SPOUT_PENDING so
> that it would not slow down or affect the present performance (any rough
> figure) ? There is no default value for this parameter right?
>
> That's a hard question as it depends on your cluster's resource and your
> processing... Having said that though, if I had no idea on both
> parameters I'd go for something like 5K or 10K and go from there (unless
> you are doing something special like producing large number of child tuples
> off a parent tuple etc).
>
> Others on this list might be able to give you a better suggestion.
>
>
>
>
>
>
> On Mon, Feb 10, 2014 at 10:34 AM, Chitra Raveendran <
> chitra.raveendran@flutura.com> wrote:
>
>> Hi
>>
>> Yeah thanks for that suggestion, killing the process was not exactly what
>> I had in mind, But just wanted to test that option too, as the whole
>> cluster was affected this weekend.
>>
>> What would be an appropriate value for TOPOLOGY_MAX_SPOUT_PENDING so
>> that it would not slow down or affect the present performance (any rough
>> figure) ? There is no default value for this parameter right?
>>
>>
>>
>>
>>
>> On Mon, Feb 10, 2014 at 3:47 PM, Enno Shioji <en...@peerindex.com>wrote:
>>
>>> T.b.h. killing your topology on a single SQL exception is prob. a bad
>>> idea, they can happen time to time and you don't want to kill your system
>>> because of that.
>>>
>>> I might look at setting TOPOLOGY_MAX_SPOUT_PENDING to some manageable
>>> number so that you won't have large numbers of tuples flying in your system
>>> and thus slashing your cluster's performance. That way when MySQL comes
>>> back up, the topology can resume processing. IMO that's much preferable.
>>>
>>>
>>> On Mon, Feb 10, 2014 at 9:53 AM, Chitra Raveendran <
>>> chitra.raveendran@flutura.com> wrote:
>>>
>>>> Thanks a lot Bijoy
>>>>
>>>> I will test that out and get back to you.
>>>>
>>>> Regards
>>>> Chitra
>>>>
>>>>
>>>> On Mon, Feb 10, 2014 at 2:14 PM, bijoy deb <bi...@gmail.com>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> You can catch the exception in your bolt (which is connecting to sql
>>>>> server) and in the catch block you can kill or deactivate the topology
>>>>> using Nimbus client,as below:
>>>>>
>>>>> NimbusClient client = .......
>>>>>
>>>>> client.killTopology(
>>>>> "topologyName");//or, client.deactivate("topologyName");
>>>>>
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> Bijoy
>>>>>
>>>>>
>>>>> On Mon, Feb 10, 2014 at 1:42 PM, Chitra Raveendran <
>>>>> chitra.raveendran@flutura.com> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> I have a Realtime data crunching pipeline which wherein storm
>>>>>> consumes data from kafka, we do some processing on this data (basically
>>>>>> counts and aggregations) and store the results into a MySql database.( I
>>>>>> use the default Kafka spout.)
>>>>>>
>>>>>> Over the weekend some network related issues led to MySQL server
>>>>>> crash, as a result storm kept re-sending the messages, the processing in
>>>>>> the storm supervisors increased drastically and CPU usage hit 100%. This in
>>>>>> turn led to some supervisors falling out of the cluster and trying to
>>>>>> restart.
>>>>>>
>>>>>> How would I be able to *handle* such an unforeseen situation? Is
>>>>>> there any way through which storm would understand that there is a MySql
>>>>>> server issue and stop re-sending data?
>>>>>>
>>>>>> Is there anyway that this exception can be caught and in *topology
>>>>>> be deactivated* for a while?
>>>>>>
>>>>>> Regards,
>>>>>> Chitra
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>>
>> Regards,
>>
>> *Chitra Raveendran*
>> *Data Scientist*
>> Mobile: +91 819753660│*Email:* chitra.raveendran@flutura.com
>> *Flutura Business Solutions Private Limited – “A Decision Sciences &
>> Analytics Company”*│ #693, 2nd Floor, Geetanjali, 15th Cross, J.P Nagar 2
>> nd Phase, Bangalore – 560078│
>>
>>
>>
>
Re: Deactivate topology incase of network issues!!!
Posted by Enno Shioji <en...@peerindex.com>.
> What would be an appropriate value for TOPOLOGY_MAX_SPOUT_PENDING so that
it would not slow down or affect the present performance (any rough figure)
? There is no default value for this parameter right?
That's a hard question as it depends on your cluster's resource and your
processing... Having said that though, if I had no idea on both parameters
I'd go for something like 5K or 10K and go from there (unless you are doing
something special like producing large number of child tuples off a parent
tuple etc).
Others on this list might be able to give you a better suggestion.
On Mon, Feb 10, 2014 at 10:34 AM, Chitra Raveendran <
chitra.raveendran@flutura.com> wrote:
> Hi
>
> Yeah thanks for that suggestion, killing the process was not exactly what
> I had in mind, But just wanted to test that option too, as the whole
> cluster was affected this weekend.
>
> What would be an appropriate value for TOPOLOGY_MAX_SPOUT_PENDING so that
> it would not slow down or affect the present performance (any rough figure)
> ? There is no default value for this parameter right?
>
>
>
>
>
> On Mon, Feb 10, 2014 at 3:47 PM, Enno Shioji <en...@peerindex.com>wrote:
>
>> T.b.h. killing your topology on a single SQL exception is prob. a bad
>> idea, they can happen time to time and you don't want to kill your system
>> because of that.
>>
>> I might look at setting TOPOLOGY_MAX_SPOUT_PENDING to some manageable
>> number so that you won't have large numbers of tuples flying in your system
>> and thus slashing your cluster's performance. That way when MySQL comes
>> back up, the topology can resume processing. IMO that's much preferable.
>>
>>
>> On Mon, Feb 10, 2014 at 9:53 AM, Chitra Raveendran <
>> chitra.raveendran@flutura.com> wrote:
>>
>>> Thanks a lot Bijoy
>>>
>>> I will test that out and get back to you.
>>>
>>> Regards
>>> Chitra
>>>
>>>
>>> On Mon, Feb 10, 2014 at 2:14 PM, bijoy deb <bi...@gmail.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> You can catch the exception in your bolt (which is connecting to sql
>>>> server) and in the catch block you can kill or deactivate the topology
>>>> using Nimbus client,as below:
>>>>
>>>> NimbusClient client = .......
>>>>
>>>> client.killTopology(
>>>> "topologyName");//or, client.deactivate("topologyName");
>>>>
>>>>
>>>>
>>>> Thanks
>>>>
>>>> Bijoy
>>>>
>>>>
>>>> On Mon, Feb 10, 2014 at 1:42 PM, Chitra Raveendran <
>>>> chitra.raveendran@flutura.com> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> I have a Realtime data crunching pipeline which wherein storm consumes
>>>>> data from kafka, we do some processing on this data (basically counts and
>>>>> aggregations) and store the results into a MySql database.( I use the
>>>>> default Kafka spout.)
>>>>>
>>>>> Over the weekend some network related issues led to MySQL server
>>>>> crash, as a result storm kept re-sending the messages, the processing in
>>>>> the storm supervisors increased drastically and CPU usage hit 100%. This in
>>>>> turn led to some supervisors falling out of the cluster and trying to
>>>>> restart.
>>>>>
>>>>> How would I be able to *handle* such an unforeseen situation? Is
>>>>> there any way through which storm would understand that there is a MySql
>>>>> server issue and stop re-sending data?
>>>>>
>>>>> Is there anyway that this exception can be caught and in *topology be
>>>>> deactivated* for a while?
>>>>>
>>>>> Regards,
>>>>> Chitra
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
>
>
> --
>
> Regards,
>
> *Chitra Raveendran*
> *Data Scientist*
> Mobile: +91 819753660│*Email:* chitra.raveendran@flutura.com
> *Flutura Business Solutions Private Limited – “A Decision Sciences &
> Analytics Company”*│ #693, 2nd Floor, Geetanjali, 15th Cross, J.P Nagar 2
> nd Phase, Bangalore – 560078│
>
>
>
Re: Deactivate topology incase of network issues!!!
Posted by Chitra Raveendran <ch...@flutura.com>.
Hi
Yeah thanks for that suggestion, killing the process was not exactly what I
had in mind, But just wanted to test that option too, as the whole cluster
was affected this weekend.
What would be an appropriate value for TOPOLOGY_MAX_SPOUT_PENDING so that
it would not slow down or affect the present performance (any rough figure)
? There is no default value for this parameter right?
On Mon, Feb 10, 2014 at 3:47 PM, Enno Shioji <en...@peerindex.com>wrote:
> T.b.h. killing your topology on a single SQL exception is prob. a bad
> idea, they can happen time to time and you don't want to kill your system
> because of that.
>
> I might look at setting TOPOLOGY_MAX_SPOUT_PENDING to some manageable
> number so that you won't have large numbers of tuples flying in your system
> and thus slashing your cluster's performance. That way when MySQL comes
> back up, the topology can resume processing. IMO that's much preferable.
>
>
> On Mon, Feb 10, 2014 at 9:53 AM, Chitra Raveendran <
> chitra.raveendran@flutura.com> wrote:
>
>> Thanks a lot Bijoy
>>
>> I will test that out and get back to you.
>>
>> Regards
>> Chitra
>>
>>
>> On Mon, Feb 10, 2014 at 2:14 PM, bijoy deb <bi...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> You can catch the exception in your bolt (which is connecting to sql
>>> server) and in the catch block you can kill or deactivate the topology
>>> using Nimbus client,as below:
>>>
>>> NimbusClient client = .......
>>>
>>> client.killTopology(
>>> "topologyName");//or, client.deactivate("topologyName");
>>>
>>>
>>>
>>> Thanks
>>>
>>> Bijoy
>>>
>>>
>>> On Mon, Feb 10, 2014 at 1:42 PM, Chitra Raveendran <
>>> chitra.raveendran@flutura.com> wrote:
>>>
>>>> Hi
>>>>
>>>> I have a Realtime data crunching pipeline which wherein storm consumes
>>>> data from kafka, we do some processing on this data (basically counts and
>>>> aggregations) and store the results into a MySql database.( I use the
>>>> default Kafka spout.)
>>>>
>>>> Over the weekend some network related issues led to MySQL server crash,
>>>> as a result storm kept re-sending the messages, the processing in the storm
>>>> supervisors increased drastically and CPU usage hit 100%. This in turn led
>>>> to some supervisors falling out of the cluster and trying to restart.
>>>>
>>>> How would I be able to *handle* such an unforeseen situation? Is there
>>>> any way through which storm would understand that there is a MySql server
>>>> issue and stop re-sending data?
>>>>
>>>> Is there anyway that this exception can be caught and in *topology be
>>>> deactivated* for a while?
>>>>
>>>> Regards,
>>>> Chitra
>>>>
>>>>
>>>
>>
>>
>>
>
--
Regards,
*Chitra Raveendran*
*Data Scientist*
Mobile: +91 819753660│*Email:* chitra.raveendran@flutura.com
*Flutura Business Solutions Private Limited – “A Decision Sciences &
Analytics Company”*│ #693, 2nd Floor, Geetanjali, 15th Cross, J.P
Nagar 2nd Phase,
Bangalore – 560078│
Re: Deactivate topology incase of network issues!!!
Posted by Enno Shioji <en...@peerindex.com>.
T.b.h. killing your topology on a single SQL exception is prob. a bad idea,
they can happen time to time and you don't want to kill your system because
of that.
I might look at setting TOPOLOGY_MAX_SPOUT_PENDING to some manageable
number so that you won't have large numbers of tuples flying in your system
and thus slashing your cluster's performance. That way when MySQL comes
back up, the topology can resume processing. IMO that's much preferable.
On Mon, Feb 10, 2014 at 9:53 AM, Chitra Raveendran <
chitra.raveendran@flutura.com> wrote:
> Thanks a lot Bijoy
>
> I will test that out and get back to you.
>
> Regards
> Chitra
>
>
> On Mon, Feb 10, 2014 at 2:14 PM, bijoy deb <bi...@gmail.com>wrote:
>
>> Hi,
>>
>> You can catch the exception in your bolt (which is connecting to sql
>> server) and in the catch block you can kill or deactivate the topology
>> using Nimbus client,as below:
>>
>> NimbusClient client = .......
>>
>> client.killTopology(
>> "topologyName");//or, client.deactivate("topologyName");
>>
>>
>>
>> Thanks
>>
>> Bijoy
>>
>>
>> On Mon, Feb 10, 2014 at 1:42 PM, Chitra Raveendran <
>> chitra.raveendran@flutura.com> wrote:
>>
>>> Hi
>>>
>>> I have a Realtime data crunching pipeline which wherein storm consumes
>>> data from kafka, we do some processing on this data (basically counts and
>>> aggregations) and store the results into a MySql database.( I use the
>>> default Kafka spout.)
>>>
>>> Over the weekend some network related issues led to MySQL server crash,
>>> as a result storm kept re-sending the messages, the processing in the storm
>>> supervisors increased drastically and CPU usage hit 100%. This in turn led
>>> to some supervisors falling out of the cluster and trying to restart.
>>>
>>> How would I be able to *handle* such an unforeseen situation? Is there
>>> any way through which storm would understand that there is a MySql server
>>> issue and stop re-sending data?
>>>
>>> Is there anyway that this exception can be caught and in *topology be
>>> deactivated* for a while?
>>>
>>> Regards,
>>> Chitra
>>>
>>>
>>
>
>
>
Re: Deactivate topology incase of network issues!!!
Posted by Chitra Raveendran <ch...@flutura.com>.
Thanks a lot Bijoy
I will test that out and get back to you.
Regards
Chitra
On Mon, Feb 10, 2014 at 2:14 PM, bijoy deb <bi...@gmail.com>wrote:
> Hi,
>
> You can catch the exception in your bolt (which is connecting to sql
> server) and in the catch block you can kill or deactivate the topology
> using Nimbus client,as below:
>
> NimbusClient client = .......
>
> client.killTopology(
> "topologyName");//or, client.deactivate("topologyName");
>
>
>
> Thanks
>
> Bijoy
>
>
> On Mon, Feb 10, 2014 at 1:42 PM, Chitra Raveendran <
> chitra.raveendran@flutura.com> wrote:
>
>> Hi
>>
>> I have a Realtime data crunching pipeline which wherein storm consumes
>> data from kafka, we do some processing on this data (basically counts and
>> aggregations) and store the results into a MySql database.( I use the
>> default Kafka spout.)
>>
>> Over the weekend some network related issues led to MySQL server crash,
>> as a result storm kept re-sending the messages, the processing in the storm
>> supervisors increased drastically and CPU usage hit 100%. This in turn led
>> to some supervisors falling out of the cluster and trying to restart.
>>
>> How would I be able to *handle* such an unforeseen situation? Is there
>> any way through which storm would understand that there is a MySql server
>> issue and stop re-sending data?
>>
>> Is there anyway that this exception can be caught and in *topology be
>> deactivated* for a while?
>>
>> Regards,
>> Chitra
>>
>>
>
Re: Deactivate topology incase of network issues!!!
Posted by bijoy deb <bi...@gmail.com>.
Hi,
You can catch the exception in your bolt (which is connecting to sql
server) and in the catch block you can kill or deactivate the topology
using Nimbus client,as below:
NimbusClient client = .......
client.killTopology("topologyName");//or, client.deactivate("topologyName");
Thanks
Bijoy
On Mon, Feb 10, 2014 at 1:42 PM, Chitra Raveendran <
chitra.raveendran@flutura.com> wrote:
> Hi
>
> I have a Realtime data crunching pipeline which wherein storm consumes
> data from kafka, we do some processing on this data (basically counts and
> aggregations) and store the results into a MySql database.( I use the
> default Kafka spout.)
>
> Over the weekend some network related issues led to MySQL server crash, as
> a result storm kept re-sending the messages, the processing in the storm
> supervisors increased drastically and CPU usage hit 100%. This in turn led
> to some supervisors falling out of the cluster and trying to restart.
>
> How would I be able to *handle* such an unforeseen situation? Is there
> any way through which storm would understand that there is a MySql server
> issue and stop re-sending data?
>
> Is there anyway that this exception can be caught and in *topology be
> deactivated* for a while?
>
> Regards,
> Chitra
>
>