You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@apex.apache.org by "Pradeep A. Dalvi" <ap...@pradeepdalvi.com> on 2016/02/08 09:27:16 UTC

Partitions in separate container?

>From my understanding, if we partition operator, they are being placed in
separate containers. Such containers might get spawned on the same or
different node.
Is it always the case that partitions get placed in separate containers? Is
this documented?
Also it would be great, if someone could point me to code snippet to
confirm the behavior?

Thanks in advance,
Pradeep A. Dalvi

Re: Partitions in separate container?

Posted by "Pradeep A. Dalvi" <ap...@pradeepdalvi.com>.

Thanks Isha.

Explanation & code snippet were very helpful.

Regards,
Pradeep A. Dalvi

On Mon, Feb 8, 2016 at 11:12 PM, Isha Arkatkar <is...@datatorrent.com> wrote:

> Hi Pradeep,
>
>     Yes, if we partition an operator it will be placed in different
> containers. Except for following 2 conditions:
>
>   1. If the partitioned operator is connected to an upstream operator by
>  CONTAINER_LOCAL stream
>   2. If the application requires more than MaxContainer(which is Max Int by
> default, but can be placed by a property). Then all Physical Operators
> after MaxContainer may get allocated in already assigned containers by
> round robin. In this case, it may happen that partitions land on same
> container due to max container resource constraint.
>
> The code for this is here:
>
> https://github.com/apache/incubator-apex-core/blob/master/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java#L350-L372
>
>
> Thanks,
> Isha
>
> On Mon, Feb 8, 2016 at 12:27 AM, Pradeep A. Dalvi <apache@pradeepdalvi.com
> >
> wrote:
>
> > From my understanding, if we partition operator, they are being placed in
> > separate containers. Such containers might get spawned on the same or
> > different node.
> > Is it always the case that partitions get placed in separate containers?
> Is
> > this documented?
> > Also it would be great, if someone could point me to code snippet to
> > confirm the behavior?
> >
> > Thanks in advance,
> > Pradeep A. Dalvi
> >
>

Re: Partitions in separate container?

Posted by Isha Arkatkar <is...@datatorrent.com>.

Hi Pradeep,

    Yes, if we partition an operator it will be placed in different
containers. Except for following 2 conditions:

  1. If the partitioned operator is connected to an upstream operator by
 CONTAINER_LOCAL stream
  2. If the application requires more than MaxContainer(which is Max Int by
default, but can be placed by a property). Then all Physical Operators
after MaxContainer may get allocated in already assigned containers by
round robin. In this case, it may happen that partitions land on same
container due to max container resource constraint.

The code for this is here:
https://github.com/apache/incubator-apex-core/blob/master/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java#L350-L372

Thanks,
Isha

On Mon, Feb 8, 2016 at 12:27 AM, Pradeep A. Dalvi <ap...@pradeepdalvi.com>
wrote:

> From my understanding, if we partition operator, they are being placed in
> separate containers. Such containers might get spawned on the same or
> different node.
> Is it always the case that partitions get placed in separate containers? Is
> this documented?
> Also it would be great, if someone could point me to code snippet to
> confirm the behavior?
>
> Thanks in advance,
> Pradeep A. Dalvi
>