You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by Yogi Devendra <yo...@apache.org> on 2016/07/25 16:41:59 UTC
Sub Partitioning the parallel partitions
Hi,
I have a DAG A->B->C.
1. A is kafka input operator reading from 4 different topics configured
with ONE_TO_ONE strategy. Thus creating 4 partitons of A.
2. B and C are configured to have parallel partitions w.r.t. their input
port. Thus, currently both B, C have 4 partitions.
I am observing B significant latency in B operator. Thus, would like to
have 2 partitions for B per partition of A. Since, application is latency
intensive, I want to avoid unifiers as far as possible.
How to achieve this partitioning?
~ Yogi
Re: Sub Partitioning the parallel partitions
Posted by Bhupesh Chawda <bh...@apache.org>.
Since the partitions of the Kafka input operator are based on (topic,
partition) combination, can we increase the number of Kafka partitions per
topic?
~ Bhupesh
On Mon, Jul 25, 2016 at 10:21 PM, Munagala Ramanath <ra...@datatorrent.com>
wrote:
> One way is to have a pass-through operator X that is parallel partitioned
> like your B currently.
> Then, connect the output port of X to B and use a suitable partitioner for
> B to create as many
> partitions as you want: A -> X -> B -> C.
>
> Ram
>
> On Mon, Jul 25, 2016 at 9:41 AM, Yogi Devendra <yo...@apache.org>
> wrote:
>
>> Hi,
>>
>> I have a DAG A->B->C.
>>
>> 1. A is kafka input operator reading from 4 different topics configured
>> with ONE_TO_ONE strategy. Thus creating 4 partitons of A.
>> 2. B and C are configured to have parallel partitions w.r.t. their input
>> port. Thus, currently both B, C have 4 partitions.
>>
>> I am observing B significant latency in B operator. Thus, would like to
>> have 2 partitions for B per partition of A. Since, application is latency
>> intensive, I want to avoid unifiers as far as possible.
>>
>> How to achieve this partitioning?
>>
>>
>> ~ Yogi
>>
>
>
Re: Sub Partitioning the parallel partitions
Posted by Munagala Ramanath <ra...@datatorrent.com>.
One way is to have a pass-through operator X that is parallel partitioned
like your B currently.
Then, connect the output port of X to B and use a suitable partitioner for
B to create as many
partitions as you want: A -> X -> B -> C.
Ram
On Mon, Jul 25, 2016 at 9:41 AM, Yogi Devendra <yo...@apache.org>
wrote:
> Hi,
>
> I have a DAG A->B->C.
>
> 1. A is kafka input operator reading from 4 different topics configured
> with ONE_TO_ONE strategy. Thus creating 4 partitons of A.
> 2. B and C are configured to have parallel partitions w.r.t. their input
> port. Thus, currently both B, C have 4 partitions.
>
> I am observing B significant latency in B operator. Thus, would like to
> have 2 partitions for B per partition of A. Since, application is latency
> intensive, I want to avoid unifiers as far as possible.
>
> How to achieve this partitioning?
>
>
> ~ Yogi
>
Re: Sub Partitioning the parallel partitions
Posted by Sandesh Hegde <sa...@datatorrent.com>.
@Yogi,
If you are up for the small experiment and share some latency number with
us, It will help improve the unifiers.
1. Avoid M*1 unifiers
2. Use the mentioned PR to launch your application with the unifiers,
https://github.com/apache/apex-core/pull/358/
On Mon, Jul 25, 2016 at 11:47 PM Ashwin Chandra Putta <
ashwinchandrap@gmail.com> wrote:
> Yogi,
> What is the reason for avoiding unifiers?
>
> Regards,
> Ashwin.
>
> On Mon, Jul 25, 2016 at 9:41 AM, Yogi Devendra <yo...@apache.org>
> wrote:
>
>> Hi,
>>
>> I have a DAG A->B->C.
>>
>> 1. A is kafka input operator reading from 4 different topics configured
>> with ONE_TO_ONE strategy. Thus creating 4 partitons of A.
>> 2. B and C are configured to have parallel partitions w.r.t. their input
>> port. Thus, currently both B, C have 4 partitions.
>>
>> I am observing B significant latency in B operator. Thus, would like to
>> have 2 partitions for B per partition of A. Since, application is latency
>> intensive, I want to avoid unifiers as far as possible.
>>
>> How to achieve this partitioning?
>>
>>
>> ~ Yogi
>>
>
>
>
> --
>
> Regards,
> Ashwin.
>
Re: Sub Partitioning the parallel partitions
Posted by Ashwin Chandra Putta <as...@gmail.com>.
Yogi,
What is the reason for avoiding unifiers?
Regards,
Ashwin.
On Mon, Jul 25, 2016 at 9:41 AM, Yogi Devendra <yo...@apache.org>
wrote:
> Hi,
>
> I have a DAG A->B->C.
>
> 1. A is kafka input operator reading from 4 different topics configured
> with ONE_TO_ONE strategy. Thus creating 4 partitons of A.
> 2. B and C are configured to have parallel partitions w.r.t. their input
> port. Thus, currently both B, C have 4 partitions.
>
> I am observing B significant latency in B operator. Thus, would like to
> have 2 partitions for B per partition of A. Since, application is latency
> intensive, I want to avoid unifiers as far as possible.
>
> How to achieve this partitioning?
>
>
> ~ Yogi
>
--
Regards,
Ashwin.