STORM with MYSQL optimizations

[masoom alam <ma...@gmail.com>]

Dear All,

Have any body worked on the configurations/optimizations needed generally
for using STORM with MYSQL. Our scenario stores data in MYSQL tables, but
as the data rate increases MYSQL starts responding very slow (in some cases
connection refused error), resulting in DBWriterBolt to slowdown. All the
Topology is bottleneck by this issue. We cannot increase the traffic at
source beyond a certain level, the reason we noted is that sink (MYSQL) or
the bolt adjacent to sink is performing slow.

Any suggestion on how should we proceed will be highly appreciated.

Thanks.

Re: STORM with MYSQL optimizations

[Sean Allen <se...@monkeysnatchbanana.com>]

Sharding is a pain in the ass and should be avoided when possible.

If its possible, I'd look for another data store that can handle a higher
load as a cluster so you don't have to worry about the details of sharding.




On Wed, Feb 26, 2014 at 8:54 AM, masoom alam <ma...@gmail.com> wrote:

> @Sean: You are right, MYSQL is not configured to handle 1000 events per
> second. I will post the results of Batch, which is also slow in our case.I
> think we should investigate thoroughly why Batch of for example 1000 is
> also slow in our case.
>
> BTW, How easy it is to configure/implement Shards in MYSQL. Any useful
> pointers?
>
>
> On Wed, Feb 26, 2014 at 6:48 PM, Sean Allen <se...@monkeysnatchbanana.com>wrote:
>
>> Is your mysql set up to handle 1000 writes a second?
>>
>> I'm going to guess no. If that is the case then Klaus' suggestions are
>> good ones. Batch or Shard.
>>
>>
>> On Wed, Feb 26, 2014 at 8:45 AM, masoom alam <ma...@gmail.com>wrote:
>>
>>> 1000 Events per second.
>>>
>>>
>>>
>>>
>>> On Wed, Feb 26, 2014 at 6:40 PM, Sean Allen <sean@monkeysnatchbanana.com
>>> > wrote:
>>>
>>>> How much traffic exactly are you pushing at mysql before the load gets
>>>> to high and it starts to buckle under the weight?
>>>>
>>>>
>>>> On Wed, Feb 26, 2014 at 8:38 AM, masoom alam <ma...@gmail.com>wrote:
>>>>
>>>>> Dear All,
>>>>>
>>>>> Have any body worked on the configurations/optimizations needed
>>>>> generally for using STORM with MYSQL. Our scenario stores data in MYSQL
>>>>> tables, but as the data rate increases MYSQL starts responding very slow
>>>>> (in some cases connection refused error), resulting in DBWriterBolt to
>>>>> slowdown. All the Topology is bottleneck by this issue. We cannot increase
>>>>> the traffic at source beyond a certain level, the reason we noted is that
>>>>> sink (MYSQL) or the bolt adjacent to sink is performing slow.
>>>>>
>>>>> Any suggestion on how should we proceed will be highly appreciated.
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Ce n'est pas une signature
>>>>
>>>
>>>
>>
>>
>> --
>>
>> Ce n'est pas une signature
>>
>
>


-- 

Ce n'est pas une signature

Re: STORM with MYSQL optimizations

[masoom alam <ma...@gmail.com>]

@Sean: You are right, MYSQL is not configured to handle 1000 events per
second. I will post the results of Batch, which is also slow in our case.I
think we should investigate thoroughly why Batch of for example 1000 is
also slow in our case.

BTW, How easy it is to configure/implement Shards in MYSQL. Any useful
pointers?


On Wed, Feb 26, 2014 at 6:48 PM, Sean Allen <se...@monkeysnatchbanana.com>wrote:

> Is your mysql set up to handle 1000 writes a second?
>
> I'm going to guess no. If that is the case then Klaus' suggestions are
> good ones. Batch or Shard.
>
>
> On Wed, Feb 26, 2014 at 8:45 AM, masoom alam <ma...@gmail.com>wrote:
>
>> 1000 Events per second.
>>
>>
>>
>>
>> On Wed, Feb 26, 2014 at 6:40 PM, Sean Allen <se...@monkeysnatchbanana.com>wrote:
>>
>>> How much traffic exactly are you pushing at mysql before the load gets
>>> to high and it starts to buckle under the weight?
>>>
>>>
>>> On Wed, Feb 26, 2014 at 8:38 AM, masoom alam <ma...@gmail.com>wrote:
>>>
>>>> Dear All,
>>>>
>>>> Have any body worked on the configurations/optimizations needed
>>>> generally for using STORM with MYSQL. Our scenario stores data in MYSQL
>>>> tables, but as the data rate increases MYSQL starts responding very slow
>>>> (in some cases connection refused error), resulting in DBWriterBolt to
>>>> slowdown. All the Topology is bottleneck by this issue. We cannot increase
>>>> the traffic at source beyond a certain level, the reason we noted is that
>>>> sink (MYSQL) or the bolt adjacent to sink is performing slow.
>>>>
>>>> Any suggestion on how should we proceed will be highly appreciated.
>>>>
>>>> Thanks.
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Ce n'est pas une signature
>>>
>>
>>
>
>
> --
>
> Ce n'est pas une signature
>

Re: STORM with MYSQL optimizations

[Sean Allen <se...@monkeysnatchbanana.com>]

Is your mysql set up to handle 1000 writes a second?

I'm going to guess no. If that is the case then Klaus' suggestions are good
ones. Batch or Shard.


On Wed, Feb 26, 2014 at 8:45 AM, masoom alam <ma...@gmail.com> wrote:

> 1000 Events per second.
>
>
>
>
> On Wed, Feb 26, 2014 at 6:40 PM, Sean Allen <se...@monkeysnatchbanana.com>wrote:
>
>> How much traffic exactly are you pushing at mysql before the load gets to
>> high and it starts to buckle under the weight?
>>
>>
>> On Wed, Feb 26, 2014 at 8:38 AM, masoom alam <ma...@gmail.com>wrote:
>>
>>> Dear All,
>>>
>>> Have any body worked on the configurations/optimizations needed
>>> generally for using STORM with MYSQL. Our scenario stores data in MYSQL
>>> tables, but as the data rate increases MYSQL starts responding very slow
>>> (in some cases connection refused error), resulting in DBWriterBolt to
>>> slowdown. All the Topology is bottleneck by this issue. We cannot increase
>>> the traffic at source beyond a certain level, the reason we noted is that
>>> sink (MYSQL) or the bolt adjacent to sink is performing slow.
>>>
>>> Any suggestion on how should we proceed will be highly appreciated.
>>>
>>> Thanks.
>>>
>>>
>>>
>>
>>
>> --
>>
>> Ce n'est pas une signature
>>
>
>


-- 

Ce n'est pas une signature

Re: STORM with MYSQL optimizations

[masoom alam <ma...@gmail.com>]

1000 Events per second.




On Wed, Feb 26, 2014 at 6:40 PM, Sean Allen <se...@monkeysnatchbanana.com>wrote:

> How much traffic exactly are you pushing at mysql before the load gets to
> high and it starts to buckle under the weight?
>
>
> On Wed, Feb 26, 2014 at 8:38 AM, masoom alam <ma...@gmail.com>wrote:
>
>> Dear All,
>>
>> Have any body worked on the configurations/optimizations needed generally
>> for using STORM with MYSQL. Our scenario stores data in MYSQL tables, but
>> as the data rate increases MYSQL starts responding very slow (in some cases
>> connection refused error), resulting in DBWriterBolt to slowdown. All the
>> Topology is bottleneck by this issue. We cannot increase the traffic at
>> source beyond a certain level, the reason we noted is that sink (MYSQL) or
>> the bolt adjacent to sink is performing slow.
>>
>> Any suggestion on how should we proceed will be highly appreciated.
>>
>> Thanks.
>>
>>
>>
>
>
> --
>
> Ce n'est pas une signature
>

Re: STORM with MYSQL optimizations

[Sean Allen <se...@monkeysnatchbanana.com>]

How much traffic exactly are you pushing at mysql before the load gets to
high and it starts to buckle under the weight?


On Wed, Feb 26, 2014 at 8:38 AM, masoom alam <ma...@gmail.com> wrote:

> Dear All,
>
> Have any body worked on the configurations/optimizations needed generally
> for using STORM with MYSQL. Our scenario stores data in MYSQL tables, but
> as the data rate increases MYSQL starts responding very slow (in some cases
> connection refused error), resulting in DBWriterBolt to slowdown. All the
> Topology is bottleneck by this issue. We cannot increase the traffic at
> source beyond a certain level, the reason we noted is that sink (MYSQL) or
> the bolt adjacent to sink is performing slow.
>
> Any suggestion on how should we proceed will be highly appreciated.
>
> Thanks.
>
>
>


-- 

Ce n'est pas une signature

Re: STORM with MYSQL optimizations

[Klausen Schaefersinho <kl...@gmail.com>]

Hi,

if you want to stick to MySQL I see two options:

1) You could try to batch your writes. The DbWriterBolt would buffer some
writes (e.g.) thousand and write them in batch mode. This is usually much
faster. However your database is not always up to date.

2) You could try to shrad you database and have several DbWriter bolts,
each writing to its own data base. You should then use a fieldsGrouping to
ensure the data always end up in the right data base.


Cheers,

klaus




On Wed, Feb 26, 2014 at 2:38 PM, masoom alam <ma...@gmail.com> wrote:

> Dear All,
>
> Have any body worked on the configurations/optimizations needed generally
> for using STORM with MYSQL. Our scenario stores data in MYSQL tables, but
> as the data rate increases MYSQL starts responding very slow (in some cases
> connection refused error), resulting in DBWriterBolt to slowdown. All the
> Topology is bottleneck by this issue. We cannot increase the traffic at
> source beyond a certain level, the reason we noted is that sink (MYSQL) or
> the bolt adjacent to sink is performing slow.
>
> Any suggestion on how should we proceed will be highly appreciated.
>
> Thanks.
>
>
>