Number of partitions

[chiranjeevi vasupilli <ch...@gmail.com>]

Hi Team,

Please help us understand, do we need to run the operator with 2 power N
partitions (2,4,8,.. )or we can set to any number of partitions as per our
requirement.

-- 
ur's
chiru

Re: Number of partitions

[Ambarish Pande <am...@datatorrent.com>]

Hi,

There is no such binding to set the number of partitions as 2^n. It is just
that in usual cases the partitions are set to be powers of 2 so that the
load is distributed to each partition equally.
Here is an example.
Say the number of partitions is set to 3.
And you send data to respective partitions by looking at the last 2 bits of
the tuple.

PART 1 - 00
PART 2 - 01
PART 3 - 10

Where will you send the tuple with 11 as last two bits?

So one of the partitions will have to bear tuples with 11 too, leading to
load imbalance.

I hope this clears things.

Thank You


On Wed, Nov 8, 2017 at 12:45 PM, chiranjeevi vasupilli <ch...@gmail.com>
wrote:

>
> Hi Team,
>
> Please help us understand, do we need to run the operator with 2 power N
> partitions (2,4,8,.. )or we can set to any number of partitions as per our
> requirement.
>
> --
> ur's
> chiru
>



-- 

_______________________________________________________

Ambarish Pande

Associate Software Engineer

E: ambarish@datatorrent.com | M: +91-9028293982

www.datatorrent.com  |  apex.apache.org

Re: Number of partitions

[vikram patil <pa...@gmail.com>]

Hi Chiranjeevi,

No there is no such restriction on a number of partitions. But it may
create skew in tuple distribution for partitions of the operator as
few of the partitions will receive more tuples than others.

Thanks & Regards,
Vikram

On Wed, Nov 8, 2017 at 12:45 PM, chiranjeevi vasupilli
<ch...@gmail.com> wrote:
>
> Hi Team,
>
> Please help us understand, do we need to run the operator with 2 power N
> partitions (2,4,8,.. )or we can set to any number of partitions as per our
> requirement.
>
> --
> ur's
> chiru