Topology scaling, KafkaSpout - Bolt tuples distribution issues

[Yiorgos Panayiotakis <y....@gmail.com>]

We have a topology which consists of a KafkaSpout and a persistence bolt
which is used
to write in a memory database and then forward tuples to other bolts. The
issue we are facing is that the topology does not scale as expected.
Specifically when we scale our topology using 2 machines - 2 workers(8
cores - 16 parallesism units) everything  seems to scale normally. We use 2
KafkaSpouts which distribute tuples to 2 persistence bolts almost evenly
(we use field grouping based on device id between spouts and bolts).
But our problem starts when we scale to 4 machines - 4 workers. Then even
though the 4 KafkaSpouts emit almost the same number of tuples those are
not distributed to the 4 persistence bolts. In fact half of those emitted
tuples result in only 1 bolt and another bolt does not get any tuples (the
2 other get a small number o f tuples).
My question is why this is happening ? It is getting much worse if we
increase the number of executors for the persistence bolts (e.g. 8
executors result only half of the bolts get any tuples), but at the same
point when we increase the number of tasks to a large number 64 for example
then every bolt gets some tuples. Is this the normal behavior ? How can we
resolve this issue ?

Re: Topology scaling, KafkaSpout - Bolt tuples distribution issues

[Yiorgos Panayiotakis <y....@gmail.com>]

Hi John,

I use field grouping based on deviceId but the deviceId numbers are well
distributed because for the moment I generate my own load, so
I know that for instance I have 100 devices emitting 1000 events totaling
100.000 events, so I expect them to be evenly distributed.

On Thu, Oct 30, 2014 at 7:14 PM, John Reilly <jr...@inconspicuous.org> wrote:

> What type of grouping are you using to distribute the tuples to the
> bolts?  Is it possible that it is a field grouping where there is a skew in
> the values used to group?
>
> On Thu, Oct 30, 2014 at 1:47 AM, Yiorgos Panayiotakis <
> y.panagiotak@gmail.com> wrote:
>
>> We have a topology which consists of a KafkaSpout and a persistence bolt
>> which is used
>> to write in a memory database and then forward tuples to other bolts. The
>> issue we are facing is that the topology does not scale as expected.
>> Specifically when we scale our topology using 2 machines - 2 workers(8
>> cores - 16 parallesism units) everything  seems to scale normally. We use 2
>> KafkaSpouts which distribute tuples to 2 persistence bolts almost evenly
>> (we use field grouping based on device id between spouts and bolts).
>> But our problem starts when we scale to 4 machines - 4 workers. Then even
>> though the 4 KafkaSpouts emit almost the same number of tuples those are
>> not distributed to the 4 persistence bolts. In fact half of those emitted
>> tuples result in only 1 bolt and another bolt does not get any tuples (the
>> 2 other get a small number o f tuples).
>> My question is why this is happening ? It is getting much worse if we
>> increase the number of executors for the persistence bolts (e.g. 8
>> executors result only half of the bolts get any tuples), but at the same
>> point when we increase the number of tasks to a large number 64 for example
>> then every bolt gets some tuples. Is this the normal behavior ? How can we
>> resolve this issue ?
>>
>
>

Re: Topology scaling, KafkaSpout - Bolt tuples distribution issues

[John Reilly <jr...@inconspicuous.org>]

What type of grouping are you using to distribute the tuples to the bolts?
Is it possible that it is a field grouping where there is a skew in the
values used to group?

On Thu, Oct 30, 2014 at 1:47 AM, Yiorgos Panayiotakis <
y.panagiotak@gmail.com> wrote:

> We have a topology which consists of a KafkaSpout and a persistence bolt
> which is used
> to write in a memory database and then forward tuples to other bolts. The
> issue we are facing is that the topology does not scale as expected.
> Specifically when we scale our topology using 2 machines - 2 workers(8
> cores - 16 parallesism units) everything  seems to scale normally. We use 2
> KafkaSpouts which distribute tuples to 2 persistence bolts almost evenly
> (we use field grouping based on device id between spouts and bolts).
> But our problem starts when we scale to 4 machines - 4 workers. Then even
> though the 4 KafkaSpouts emit almost the same number of tuples those are
> not distributed to the 4 persistence bolts. In fact half of those emitted
> tuples result in only 1 bolt and another bolt does not get any tuples (the
> 2 other get a small number o f tuples).
> My question is why this is happening ? It is getting much worse if we
> increase the number of executors for the persistence bolts (e.g. 8
> executors result only half of the bolts get any tuples), but at the same
> point when we increase the number of tasks to a large number 64 for example
> then every bolt gets some tuples. Is this the normal behavior ? How can we
> resolve this issue ?
>