Re: Partitioning information and current operators Id in setup()/activate() method.

[Thomas Weise <th...@apache.org>]

Implementing Partitioner should cover this. The partitioner is called when
constructing the physical plan (at initialization) and then, in the case of
dynamic partitioning, on demand.

As part of the partitioning logic you can set properties on the operator
instances that are specific to the partition. An example for this is the
Kafka input operator, where each Apex partition will be setup with the
Kafka topic partition(s) it is responsible for.

Thanks,
Thomas


On Thu, Sep 21, 2017 at 6:36 AM, Ananth G <an...@gmail.com> wrote:

> Hello All,
>
> I was wondering if there is a good way to get the following two values in
> an operator when the operator is in the activate/setup methods
>
> 1. The total number of operator physical instances that are going to be
> launched
> 2. The ordinal position of the current operator in the current set of
> physical operators
>
> I am looking for this information so that I can implement a logic in the
> operator wherein each operator takes a part of the responsibility . I can
> get the above values when the dynamic partitioning interface is implemented
> and the call back happens to the assign() method. But the issue is that the
> assign() method does not seem to be invoked at the beginning but only when
> the dynamic partitioning is being done ( In other words the assign() method
> is not called until a certain time has passed since the beginning of the
> launch of the operator. I would like the above values when the operator is
> starting in the activate()/setup() methods.)
>
> Any advice as to how these values can be obtained ?
>
> Regards,
> Ananth

Re: Partitioning information and current operators Id in setup()/activate() method.

[Ananth G <an...@gmail.com>]

Thanks a lot Thomas. It is indeed working as you mentioned. It was a bug in my code that was present in an edge condition/use case of a single Apex operator configuration. Due to the buggy code in the partition method when the number of Apex operators was configured to be one,  I was returning back the prototype operator ( the original instance ) without setting some operator specific internal flags. This was misunderstood by me as that the partition logic did not get triggered in the first place. 

Thanks for confirming.


Regards,
Ananth
> On 22 Sep 2017, at 1:33 am, Thomas Weise <th...@apache.org> wrote:
> 
> Implementing Partitioner should cover this. The partitioner is called when
> constructing the physical plan (at initialization) and then, in the case of
> dynamic partitioning, on demand.
> 
> As part of the partitioning logic you can set properties on the operator
> instances that are specific to the partition. An example for this is the
> Kafka input operator, where each Apex partition will be setup with the
> Kafka topic partition(s) it is responsible for.
> 
> Thanks,
> Thomas
> 
> 
> On Thu, Sep 21, 2017 at 6:36 AM, Ananth G <an...@gmail.com> wrote:
> 
>> Hello All,
>> 
>> I was wondering if there is a good way to get the following two values in
>> an operator when the operator is in the activate/setup methods
>> 
>> 1. The total number of operator physical instances that are going to be
>> launched
>> 2. The ordinal position of the current operator in the current set of
>> physical operators
>> 
>> I am looking for this information so that I can implement a logic in the
>> operator wherein each operator takes a part of the responsibility . I can
>> get the above values when the dynamic partitioning interface is implemented
>> and the call back happens to the assign() method. But the issue is that the
>> assign() method does not seem to be invoked at the beginning but only when
>> the dynamic partitioning is being done ( In other words the assign() method
>> is not called until a certain time has passed since the beginning of the
>> launch of the operator. I would like the above values when the operator is
>> starting in the activate()/setup() methods.)
>> 
>> Any advice as to how these values can be obtained ?
>> 
>> Regards,
>> Ananth